Andrew Gelman has a brief post up on his blog comparing the way bug reports in open source software are received to the way many researchers respond to criticisms of their work. The comments there are good, and cover my first reaction, which was, “Developers respond well to bug reports?” But that’s a bit tongue in cheek. I do think that, overall, developers are a bit more responsive to bug reports than scientists are to published criticisms of their work. Here are my theories as to why that is:
- Bug reports are not analogous to published criticisms. Bug reports are the primary way for people to give feedback to the maintainers of a project, while historically, much criticism of research has been through less formal mechanisms such as email, questions at conference or post talks, and face-to-face at lab meetings, conferences, etc. I have never seen a scientist respond poorly to a critique at a lab meeting or a poster session, for instance. If you factor in these other interactions, the average emotional response to criticism might be more equal.
- Research careers are measured in papers. The academy clings to its traditions, and that’s one of the big ones. Papers are difficult to amend in significant ways, and they take a lot of work to produce. If software developers could only push to production once every year or so, I bet we’d find receiving them much more stressful!
- Most bug reports don’t existentially threaten software projects. I mean, I’m sure it’s happened (and now I’m kind of curious when and how!). But many critiques of published research are suggesting that reported effects may not exist, which is a much bigger blow than “hey your app keeps freezing”. So long as researchers are lauded not for the quality of the work they do but the significance of the results they achieve, this kind of critique is going to feel like a threat, especially to non-tenured researchers and scientists in other types of precarious positions.
If I have one critique of the open science movement, which I otherwise endorse and consider myself a part of, it’s that we focus too much on the behavior of individual researchers and not enough on the systems which motivate that behavior. It’s not that developers are better people than scientists. It’s that the systems developers operate within are set up to reward and punish different things.