Assessing Quality in Media Research

A few months ago, I started taking notes for a post titled “Quality Should Not Be a Dirty Word.” This was initially prompted by reading that Ed Norton (an actor whom I like) would be starring in the next Hulk movie (a franchise I think could be fun), but that the movie would be directed by the fellow who did The Transporter 2 (a fairly abysmal movie). The disappointment I experienced made me want to write a blog post, and it seemed geeky enough to fit in here, but then I realized that it seemed somewhat out of bounds for an academic blog: Media researchers aren’t supposed to make evaluative judgments like this. That kind of reaction is for fans—though, when you think about it, it’s not like media researchers’ tastes don’t influence what they write about. Thoughts of the Hulk behind me, I suddenly started taking notes on the relative lack of research and reflection on the how aesthetic standards are formed and applied, including by academics (at least since Bourdieu closed the book on it for many since he described taste in terms of class values).

I saved the post as a draft and never really got back to it. The argument in my notes, anyway, was this paragraph:

Perhaps our goal, then, should not be to calculate a universal rubric for quality or engagement, which results in theories that seem reductive, overreaching, or awkwardly cobbled together, like the (admittedly interesting and ambitious) GameFlow model does for games. Rather, we should identify what it is that individuals and communities find rewarding about their favorite works. Not, “Why is Star Wars so good?” but “What is it that people find good about Star Wars?” Closely related to this is the question, “What do people do with star wars?”

I ended up following up on this idea somewhat in my recent post on how we should evaluate “art” in games, suggesting that we break our evaluation down into formal elements (like focusing on “atmosphere” versus “storytelling” in games) and practical results (like lasting historical relevance, or inspiring immediate political action).

Outside of games, another specific example for this (to be described in the aborted essay) was the example of “worldbuilding,” which Cory Doctorow describes as “the science fiction writer’s practice of meticulously (or not) drawing up the contours of a fictional place—from the physics to the biology.” Cory’s not a fan of this practice. An essay he links to on this topic suggests that M. John Harrison is not a fan either; he calls worldbuilding “the great clomping foot of nerdism. It is the attempt to exhaustively survey a place that isn’t there. A good writer would never try to do that, even with a place that is there.” And, actually, I’m not a big fan of reading other people’s worldbuilding either (though I recall from my high school days that worldbuilding was the most fun part of designing homemade roleplaying games). Nevertheless, we must recognize that for some, at least, worldbuilding not only holds certain appeal as a formal element, but may be evaluated as effective or ineffective by audiences. We can see differences in fannish appraisal of quality in worldbuilding when we contrast responses to how this was handled in the original Star Wars trilogies and the material that followed with the prequels, or in viewing fan reaction to different Tolkien books or the comparison between the books and the movies. Here, discussion of quality is not monolithic and oppressive, defined by the wealthy classes and handed down from on high, but posed in context, framed by questions of use: quality for whom and according to what? Socioeconomic class clearly plays a part, as Bourdieu discusses, but that may not tell the whole story of how taste is made in more specific instances.

There you have most of my notes for “Quality Should Not Be a Dirty Word” (minus some additional babbling about Stuart Hall and jazz). I’m not going to finish that essay, though, because I have much else to do, and, more importantly, because Jason Mittell already did a fine job of it and posted it to his blog. Lost in a Great Story: Evaluation in Narrative Television (and Television Studies)” not only reflects on media researchers’ shyness about issues of evaluation, but attempts to apply a culturally framed evaluation to a particular media object, the television series Lost. He gets away from the all too typical academic approach to television which cloaks author opinion with terms like “complexity” or “nuance” (and is dead on in pointing out this practice), and instead suggests some specific formal factors that keep people coming back to the show.

I do think the analysis could go a bit further. A noteworthy counterargument comes in the comments that follow from Ian Bogost, who suggests that fans’ faith (that all the plot twists will fit together as if planned) seems is quite likely to be misplaced. (Though I disagree that leaving giant questions lingering between seasons is a way of fleecing suckers in the audience, which I find analogous to arguing that each chapter of a book should leave no major questions dangling). More niggling to me is the question of why I should accept the argument that Lost‘s use of the techniques described here is superior to Heroes‘s use, which seems attributed to some unidentified mass of fans and critics. Overall, though, I think this is a noteworthy step toward discussing issues of quality assessment in media research, and I’m sure glad some other Jason took the time to do a good job on it.

3 thoughts on “Assessing Quality in Media Research

  1. I have to disagree with Cory st al on the importance of World Building. While I’m sure it can be overdone, the basic practice is obviously necessary. (Whether or not that makes interesting reading is up to the reader, though. I’ll note that I was the kind of kid who would buy wargames just to read the rules. )

  2. Just wanted to thank you for the kind words! I didn’t try to tackle the issue of negative evaluation, in part because I was writing for an anthology focusing on Lost & thus didn’t want to gaze too closely at other shows. But your Heroes comparison is important – we need to make arguments as to why some creative work succeeds and why others fail. I am more reluctant to engage with negative evaluation, as it can slip into elite castigation of programs that other people find valuable – sure, Lost-haters will disagree with my assessment, but they’ll be less offended than Heroes-lovers reading a sustained attack on the show. In some earlier, more casual blog posts, I have offered some negative evaluation of Heroes , The Nine, and even Lost. Do these address your niggles?
    -Some other Jason

  3. Church:
    I’m sure Cory would agree that you need to give readers/audiences a sense of setting in any story, but I think the “worldbuilding” they’re talking about is when authors focus on the details and physics of a world for its own sake. He describes settings himself, of course, as in Down and Out in the Magic Kingdom, but only insofar as it seems necessary to establish theme, tone, and parallels to our own present world. (And, actually, I think there’s an argument to be made that the original Star Wars trilogy was more well-received than the prequels in part because it gave us just enough of the setting to set the stage and tone, rather than intending to fill in gaps that fans were quite happy to do in the ensuing years.)

    Thanks for the comment! I’ll certainly not hold it against you for staying on-point when writing a piece for an anthology about one specific show. And yes, that post about Heroes in particular does help clarify how one might evaluate two different shows in terms of one of the formal characteristics you mentioned. Perhaps it’s just a function of so much still being up in the air with Lost, but it does feel more like people discussing that show are more concerned with “what will happen” than “what just happened,” whereas the chatter on the Heroes forums (especially following the first time travel episode) suggest that its viewers struggle to make sense of the latter question as much as the former.

    Updating this comment after reading Jason’s Lost post linked above: Man, and I felt so clever for the “what will happen” vs. “what just happened” distinction. No, you’re right, “why is this happening” is a better term for the former. But my point was just that Heroes seems to leave people working doubletime to fill in plot holes and seeming inconsistencies. I can think up reasons why Peter couldn’t just fly away on his own if he goes into “bomb mode,” but I probably shouldn’t have to be the one doing that work.

Comments are closed.