Along with most of the rest of the world, in-person testing for me stopped abruptly in March, 2020. I was about to launch piloting for the final project of my PhD, an electroencephalography experiment studying timing and speech perception – a logistical impossibility in locked down London, both then and now. It was a quick and painful pivot to online behavioural work via Gorilla, but I have to say that it’s been a lot of fun. It’s hard to imagine going back to onerous recruitment, booking people in, taking out cash advances that will take the university months to reimburse me… Online testing is fast, convenient, and easy to implement in general. But there have also been some difficulties, one of which I would like to bring to light, especially for those of us who research speech, music, or run multisensorial and/or temporally dynamic tasks.Continue reading Your Online Sample May Be Mostly Gamers
In 2010, Henrich, Heine, and Norenzayan produced an insightful, if damning paper exposing the extreme ethnic, cultural, geographical, and social class biases that pervade most psychology experiments. The authors estimate that virtually all participants are ‘Western, educated, and from industrialized, rich, and democratic countries’, or ‘WEIRD’. As the authors point out, nobody wants to publish in the ‘Journal of Personality and Social Psychology of American Undergraduate Psychology Students’, but even today, few scientists have the grounds to claim their findings generalise across socio-cultural-experiential barriers. And despite a somewhat non-establishment disciplinary foundation, music science is just as vulnerable to selectively representative samples as the next field. Nonetheless, diversifying the subject pool is increasingly recognised as a problem, and one that won’t go away on its own.
But what about the rest of the scientific process? In scientific experiments, we often force choices, or attempt to reveal contrasting features between supposedly objective categories – but do our designs reflect something factual, a physical constant, or a belief we hold because, well, everyone else we know believes the same thing? Does the research question itself reflect a universal human experience, or just ours? I don’t think that diversity in the recruitment process is enough. Rather, considering scientific questions from another cultural (or experiential) lense permits the space and detachment we need to determine what is truly ‘objective’ – or, at least, what amounts to a shared subjectivity across human societies.
This isn’t just an issue for the so-called ‘soft science’ of psychology. Take the concept of species. It doesn’t sound like a topic that biologists would do much philosophizing over, until you examine the term in its historical context, and adopt a data-driven, rather than ‘common-sense’ approach. As Darwin put it, the status of ‘species’ is “one arbitrarily given for the sake of convenience to a set of individuals closely resembling each other”, a sentiment that has only gained credibility as time passes, according to the Smithsonian:
The idea of a species is ultimately a human construct. With advancing DNA technology, scientists are now able to draw finer and finer lines between what they consider species by looking at the genetic code that defines them. How scientists choose to draw that line depends on whether their subject is an animal or plant; the tools available; and the scientist’s own preference and expertise.
By the way, this also applies to hard sex/gender and racial categories; in short, they can be useful—alternatively, destructive, and sometimes deadly—for quotidian communication, and capture or drive many of our social structures and group identities. But these classifications are far from universal or elemental, and should be handled with care in a scientific setting, particularly if you don’t take factors such as culture, experience-driven neuroplasticity (e.g., a lifetime of performing the social role of ‘female’ may unsurprisingly make your brain look more ‘female’), and epigenetics into account when explaining between-group differences. And let’s not forget the effects an experimenter can introduce by unintentionally treating her subjects differently, as a result of her unconscious bias and/or personal stake in a specific experimental outcome.
Anyways. Music science. Continue reading Cross-Cultural Validity is a Philosophy of Science Problem