Appropriate empirical evidence?
An image of a man who is definitely not a college student (certainly not traditionally aged) accompanies an article called “Men Assume Sexual Interest When There May Be None†in a recent piece by a HealthDay reporter, a piece that’s been published on various Web sites. (In case of link rot, I’ve placed a screen shot here.)
In the sixth paragraph of the piece we find out that the study is based on 43 male and 43 female college students aged 18-22. That is the only part of the article where the participants are referred to as college students. Otherwise, the entire piece is about the behavior of men and women generally speaking.
There are several fields that base a good chunk of their empirical research on studies of students.* This is usually done due to convenience. And perhaps regarding some questions, age and educational level do not matter. But the issue is rarely addressed directly. In many instances it seems problematic to assume that a bunch of 20-year-olds in college are representative of the entire rest of the population. So why write it up that way then? At best, in the conclusion of a paper the authors may mention that future studies should/will (?) expand the study to a more representative sample, but these studies rarely seem to materialize.
This is one of my biggest pet peeves when it comes to certain types of scholarship. And I do mean scholarship. Because it is not just the journalistic reports that make the leap. The academic articles themselves use that kind of language. It is part of a larger question that’s been of interest to me for a while now: Historically, how have various fields settled on what is acceptable empirical evidence in their domain and what are the appropriate modes of analysis? Papers that get into top journals in one field wouldn’t even make it off the editor’s desk for review in another field due to the data and methods used. But then when it comes to reporting findings to the public, it all becomes one big general pool of work where the methods and the validity of the findings don’t seem to matter anymore.
[*] Note that recently I have been doing studies on college students myself. First, I have a concrete substantive reason for doing so (they are the most highly network-connected age group, which helps to control for regular use). Second, when I write up the work, I never draw huge generalizations about all users. I always report on “college students†or “study participantsâ€. I do not simply conclude that whatever I find about college students is representative of all Internet users. It would be wrong to do so.
July 3rd, 2006 at 5:40 pm
This is a really interesting topic and one that I’ve been thinking about lately too. Given my field, I tend to think about nationally representative samples when confronted with any reserach question, though of course they are not always possible. bu tthere are some fields where it is common practice to assume that all hmans are interchangeable, because what is beign studied is a biological phenomenon. Psychology (where almost ALL research is based on first years enrolled in PSYCH101) and medicine are the fields that come to mind first.
The problem is that the reserachers cannto evaluate their assumption that all people everywehere work the same, because they just do not have data that data to do that evaluation, i.e. they have only data from college first years, or people under treatment at major research hospitals).
I could go on, but i’ll see if this thread stimulates interest first.
July 3rd, 2006 at 5:47 pm
Thanks, Steph, it’s not surprising that the topic would resonate with you. As you say, there is not much basis for the assumptions those studies make, that’s partly what is so frustrating. And as I noted, in the best of cases, the most you’ll see is a brief acknowledgement about problems of generalizability at the end, but nothing to address them. And the entire article is still written in language that generalizes.
There is a bit of discussion over on Crooked Timber, and there were some comments over at Political Animal as well, after Kevin Drum picked it up. But careful with some of the latter, some of those comments are incredibly frustrating.