The constitution of knowledge is cross-disciplinary

This old tweet has recently been making the rounds and sparked up some discussion among my Facebook friends:

As someone with a background in both science and the humanities, I am continually exhausted by the antagonism between them.  In any given argument I usually side with humanities advocates, because STEM workers are way more likely to be dismissive dicks.  But this critique really misses the mark.

All disciplines ought to teach about how knowledge is constituted in their domain.  That’s as true for biology and math and computer science as it is for history and philosophy and art.

It’s true that some of the disciplines most likely to discuss knowledge constitution are in the humanities.  Philosophy has epistemology, for instance, and there are subfields of history and sociology focused on knowledge constitution.  But in STEM we have statistics, and subfields of psychology and computer science concerned with what is knowable and how we know it.

Regardless, I don’t agree that we can or should assign ‘knowledge constitution’ to specific fields.  All knowledge is actively constructed and we ought to be teaching people about that process even as we’re teaching the current results of the process.  Which is, I think, what Neil Degrasse Tyson is getting at.

 

A matter of trust

This originated as a post to a mailing list on the subject of blockchains and how they might help the cause of open science.  The quote below is the claim I was directly responding to.

Shauna: “but the scientific community is arguably the most effective trust-based system in human history” – according to this view we wouldn’t need version control systems or preregistration either. I couldn’t disagree more; trust has no place in science. To me, that’s one of the major things with open scientific practices: removing the trust from science and instead practice transparency.

My response:

Trust is a fundamental issue in all human relationships and communities.  Every single interaction we have with other human beings involves some level of trust.  Just today I had to trust the people who harvested and packaged my food not to have accidentally or maliciously poisoned me, the drivers on the street to obey traffic conventions, and my house mate to have locked the house and not invited anyone dangerous inside — and those are only the most obvious examples.  I could come up with dozens more.

Different situations require different levels of trust.  If a situation requires more trust than you currently have, you can try to increase trust in a number of ways.  You can build new technologies, but you can also strengthen relationships, create neutral institutions, or add legal or regulatory force to your agreements.  None of these work perfectly, and often you’re best off pursuing a combination of them.  In all cases, though, you will have to trust someone at some point – it’s just a matter of deciding which system will allow you to trust in a way that’s acceptable to you.

The scientific community has trust issues, yes, like every other human community.  But its trust issues are of a specific type.  When you read a scientific paper, what makes you doubt the findings?  Personally, I’m not worried that the authors have faked the data, or that the publisher has changed the content of the paper without anybody knowing, or that the paper is stolen or plagiarized.  I know that the scientific community has very strong norms against these types of violations, and so they’re relatively rare.  Broadly speaking, I trust the scientific community to minimize these problems.  There’s not a lot of communities I would trust like that, which is why I claimed that science is special in this way.

The trust issues that the scientific community currently has are largely based around mis-aligned incentives.  I trust most scientists not to engage in outright fraud but I don’t trust them not to make choices in their research practices that may hurt their careers.  They know how the funding, publication, and tenure systems work, and they know that replications, preregistration, and following strict practices to minimize false positives will hurt their careers.  Simply put: most scientists don’t trust that taking actions to make science better will be rewarded rather than punished.  In a world of decreasing funding and a decaying social safety net, is anyone surprised that people do what’s best for themselves within the existing norms of the community?

My focus, then, is on supporting initiatives that help scientists trust that taking actions to make science better will be rewarded rather than punished.  I don’t see how blockchain helps with that even slightly.  I’d rather put time and energy and resources into things like lobbying funders to require certain research practices, supporting journals that facilitate preregistration and minimize publication bias, convincing departments to require a minimum number of replications per researcher per year, and educating students and early career researchers about the importance of these practices.  In other words, changing the norms so that engaging in these behaviors is easy rather than hard – because I trust humans to prefer the easy thing to the hard thing.

The yin and yang of questions and answers

In a previous post, I wrote that asking questions is harder than answering them, although I qualified that in a big way with “answering [questions] involves going back over and over again and updating our hypotheses, which makes answering questions feel hard”.  I want to revisit this claim.

Some of you may be familiar with the “reproducibility crisis” happening in the sciences, where many popular and well-known results have failed to replicate.  But what does failure to replicate mean?

Maybe it means that there was something wrong with the original study.   Maybe it means that there was something wrong with the replication.  But those aren’t the only options.  As nobel laureate psychologist Daniel Kahnamen wrote in an open letter to the scientific community:

In the myth of perfect science, the method section of a research report always includes enough detail to permit a direct replication. Unfortunately, this seemingly reasonable demand is rarely satisfied in psychology, because behavior is easily affected by seemingly irrelevant factors.

Note that underspecification of methods is an issue in all sciences.  Psychology just has a particularly rough time of it because psychology itself, like other soft sciences, is so underspecified.  Behavior is affected by seemingly irrelevant factors which are actually relevant previously unspecified factors.

In a better world, replication would be a collegial and common process involving many back-and-forths between originators and replicators.  Each replication could help identify new factors that turn out to be surprisingly relevant.  Eventually the hypothesis and methodology would be specified enough to permit consistent replication, at which point we’d have both our question and our answer.

This example makes clear that asking and answering questions are not two separate activities.  They are intertwined, at least when the questions and answers are new.  So it makes no sense to say, “Asking questions is harder than answering them” or vice versa, because you can’t do one meaningfully without also doing the other.

FYI: to read more about replication, try this article I wrote back in 2014 on the Open Science Collaboration blog: What we talk about when we talk about replication.

 

Hard and soft sciences

Back when I was a research scientist, I straddled the boundary between “hard” and “soft” sciences.  I did social psychology, which is a pretty soft science as sciences go, but I paired it with biology and physiology in general and endocrinology in particular, which meant getting a taste for some of the harder stuff.

I have never particularly liked the terms “hard” and “soft”, though, because it’s too easy to conflate them with “hard” and “easy”.  There’s a saying that goes: the soft sciences are easy to do poorly and hard to do well.  They are easier to do poorly than the hard sciences, and harder to do well than the hard sciences.  Here, have a chart:

What’s going on here?  The hard sciences are better developed than the soft sciences, so it’s clearer when someone’s making obvious mistakes, cutting corners, or making under-supported claims.  That makes it difficult to do poor work.  It’s also difficult to good work, of course.  The easiest thing to do in the hard sciences is to meet a minimum level of competency and do solid but uninspiring work.

Meanwhile in the soft sciences there’s questions even about the field’s basics.  There’s still a minimum level of competency, but it’s much less stringent than in the hard sciences.  So sloppy researchers tend to end up in the soft sciences.

Here’s another way to approach the hard/soft distinction. What’s easier, formulating questions or answering them?  It’s almost always easier to do the latter, provided you’ve very clearly and specifically formulated your question.  Of course we seldom do get our questions right on the first try, and so answering them involves going back over and over again and updating our hypotheses, which makes answering questions feel hard.  But the hardest parts of answering questions are really secretly still about asking them.

In the hard sciences, it’s easier to clearly and specifically formulate questions because so much knowledge has already been established.  Isaac Newton famously said (paraphrased) ‘If I have seen further it is only by standing on the shoulders of giants.’  The hard sciences are full of giants, with shoulders for modern researchers to stand on.  The soft sciences are by and large still on the ground.

For this reason, I prefer the terms “developed” vs “undeveloped” sciences.  I think it comes closer to the essential difference.

Note: this post has an update/correction post.

Science vs Software

Andrew Gelman has a brief post up on his blog comparing the way bug reports in open source software are received to the way many researchers respond to criticisms of their work.  The comments there are good, and cover my first reaction, which was, “Developers respond well to bug reports?”  But that’s a bit tongue in cheek.  I do think that, overall, developers are a bit more responsive to bug reports than scientists are to published criticisms of their work.  Here are my theories as to why that is:

  • Bug reports are not analogous to published criticisms. Bug reports are the primary way for people to give feedback to the maintainers of a project, while historically, much criticism of research has been through less formal mechanisms such as email, questions at conference or post talks, and face-to-face at lab meetings, conferences, etc.  I have never seen a scientist respond poorly to a critique at a lab meeting or a poster session, for instance.  If you factor in these other interactions, the average emotional response to criticism might be more equal.
  • Research careers are measured in papers.  The academy clings to its traditions, and that’s one of the big ones.  Papers are difficult to amend in significant ways, and they take a lot of work to produce.  If software developers could only push to production once every year or so, I bet we’d find receiving them much more stressful!
  • Most bug reports don’t existentially threaten software projects.  I mean, I’m sure it’s happened (and now I’m kind of curious when and how!).  But many critiques of published research are suggesting that reported effects may not exist, which is a much bigger blow than “hey your app keeps freezing”.  So long as researchers are lauded not for the quality of the work they do but the significance of the results they achieve, this kind of critique is going to feel like a threat, especially to non-tenured researchers and scientists in other types of precarious positions.

If I have one critique of the open science movement, which I otherwise endorse and consider myself a part of, it’s that we focus too much on the behavior of individual researchers and not enough on the systems which motivate that behavior.  It’s not that developers are better people than scientists.  It’s that the systems developers operate within are set up to reward and punish different things.