LOGO
In this video, Dr. Femke Truijens explains the importance of understanding the meaning behind the numbers.
This video was shown at the annual conference of the Society for Psychotherapy Research, at Trinity University in Dublin, Ireland.
Learn more about the activities and findings of the
Meaningful Measurement (MEANS) Lab.
?
Read the research program here, or contact Femke Truijens (PI MEANS Lab)
In mental health and clinical research, much is measured: diagnoses, effects of interventions, routine outcome measurement (ROM), group comparison, and so on. Typically, we ask people to score themselves on validated self-report questionnaires. Validation is often seen as a guarantee that the figures can also be interpreted and compared in a valid way. Personal variation is “averaged out” in it, and outliers are brushed off as “noise” or error.
But is that justified? What do people actually mean when they quantify their own experiences in a pre-structured framework like a questionnaire? Do their scores capture the meanings that we as researchers or clinicians also attach to them? And are those numbers indeed meaningful and validly comparable?
The premise in this line of research is that simply assuming validity of figures because they are collected with validated questionnaire relies on face validity. It requires thorough research into scoring processes and personal meaning-making of participants to investigate whether “the numbers” are indeed valid representations that can also serve as valid “data” for research and clinical monitoring.
Questionnaire scores as words in a clinical narrative
This line of research focuses on the meaningfulness of numbers within a personal narrative and meaning framework. In it, personal meaning is not dismissed as noise but rather examined as a potentially vital part of a personal narrative. I examine processes of meaning formation and translation(meaning-making) that people go through when they score questionnaires, and conversely the effect that that process of meaning-making and formation has on the variables and (clinical) processes actually reported. Such a scoring process can be transformative : scoring can change interpretations, which on the one hand affects the validity of data, but on the other hand can be a particularly informative and meaningful process for clinical research and real-world applications.
Numbers are words too
– that is the premise in the “meaningful measurement” line of research. In this line of research, I take a hermeneutic perspective on shaping and interpreting figures in (research into) clinical practice. I work on three lines of research:
- Meaning-making processes: mixed method research on scoring and scoring as part of a clinical narrative. I look at scoring tendencies, -obstacles, and interpretations: experimental research on interpretation of items and questionnaires as a whole, with attention to the message people want/think to convey through a questionnaire, and the “hearer” they assume in it. In this line, I use “Thinking Aloud” and interviews.
- Response Shifts: when people begin to interpret items on questionnaires differently – via recalibration, re-conceptualization or reprioritization – as a result of an intervention (e.g., therapy), there is a response shift. I examine the occurrence of this phenomenon through qualitative methods, and whether response shifts offer opportunity for a new, positive outcome assessment.
- Validity theory: In psychology, we standardly conceive of validity as “measuring what an instrument intends to measure. This definition is tied to how valid the data can be interpreted. Although such validity is often seen as a “property” of an instrument itself, the validity literature also argues that validity of interpretations is not given by the instrument, but depends on the context, objectives, administration conditions and users (both respondents and test takers). Whereas “validity” is often seen as a somewhat narrow term, users of questionnaires do validation throughout the process, such as checking test conditions and verifying that the respondent understood the items correctly. This method of validation-in-action is diametrically opposed to the standard view of validity, even though it is very much in line with practice. In“the great validity debate,” this approach is called the user-oriented or the argument-based approach. In this line of research, we explore how users of self-report measures normally validate, and how we can further develop the validity toolbox to support users in this process.
Qualitative research on quantification in self-report measures
In the MEANS Lab, we explore how to use qualitative research methods to bring out the story behind the numbers.
In the “I feel 4 out of 5 Depressed” study, we compare the following qualitative methods:
- Thinking Aloud Methods literally ask the respondent to think aloud while scoring the questionnaire. By (audio)recording this verbalization, we gain insight into active meaning-making, e.g., interpretations of items and evaluation of response options. But the scoring process itself also becomes insightful in this way, for example, if a respondent is in doubt, often has difficulty understanding items, becomes emotional (measurement reactivity), starts taking over word choices (performativity of measurement instruments).
- Cognitive Interview Methods ask the respondent after scoring how they look back on that process. In the “I feel 4 out of 5 depressed” study, we first ask semi-structured questions about one’s own view and experience of depression, then go item by item through the scored questionnaire and ask meaning-making during scoring. In the study, we compare how insightful the data from the two methods is and how they help validate scores; as well as how the methods are perceived and how they might be applicable in practice. For example, consider a clinician who has ROM questionnaires filled out and wants to know if she should sit next to them, or has the questionnaire filled out in the waiting room and asks at the start of the session how it went.
In addition, the MEANS Lab is working with McGill University ‘s Data Ethics Chair to develop a low-threshold method of collecting qualitative meta-data . These meta-data should be collected in such a way that they are no put additional pressure or burden on users, but well Allow the user to indicate what or if something is going wrong or difficult during data collection (think response options that do not describe the experience well, large discrepancy in scores in different contexts, or opposite interpretation of surveyed concepts). On the one hand, this meta-data allows the test taker to identify validity problems in the standardized data collection. On the other hand, this meta-data can be used to develop bottom-up new hypotheses about where, when, with whom, and under what circumstances standardized questionnaire scoring goes non-standard.
Questionnaire scoring as a Hermeneutic Circle
In the Meaningful Measurement (MEANS) Lab, we start from a
hermeneutic approach
. This philosophy, elaborated by H.G. Gadamer and others, assumes that meaning is not embedded in objects or constructs, but is always interpreted by the user or viewer. Each user does so from their own pre-understanding, based on their own knowledge and experience and shared collective frames of meaning. From those shared frames of meaning, we learn all kinds of things that we take as given, or truth. But we as humans also give it our own color, linking our own experiences and associations to those shared meanings. As a result, although we can understand each other broadly, there may be differences in interpretation. As a result, people are constantly probing and aligning meanings of themselves and others. This ongoing exchange is called the hermeneutic circle between signifiers and their interpretations.
In the MEANS Lab, we assume that questionnaires are also interpreted by users, from shared frames of meaning but also by their own (pre)understanding. For example, the questionnaire was developed from a (shared) frame of meaning, but each respondent (patient/client/caregiver) has to interpret it from their own frame of meaning in order to score, on which scores are then interpreted again by the test taker (e.g., diagnostician or practitioner).
This makes questionnaire scoring a
meaningful
,
dynamic, interactive
and
changeable
act of signification and interpretation.
MEANINGFUL MEASUREMENT therefore calls for:
- Iterative process betweenstakeholders‘ meaning-making, interpretation and validation
- Explicit assumptions, goals, interpretations and consequences à reflexivity and validation-in-action
(cf., argument-based validity) - Tracking qualitative/process meta-data (bottom-up meaningful measurement)
- Test-user who remains curious and sensitive about who or what does not fit the norm/average
- Gathering evidence for what is not (yet) evident
KOG Members with expertise in Meaningful Measurement:
Click on a tile to delve deeper into the next theme:
© Copyright – Kwalitatief Onderzoekscollectief GGZ