The yin and yang of questions and answers

In a previous post, I wrote that asking questions is harder than answering them, although I qualified that in a big way with “answering [questions] involves going back over and over again and updating our hypotheses, which makes answering questions feel hard”.  I want to revisit this claim.

Some of you may be familiar with the “reproducibility crisis” happening in the sciences, where many popular and well-known results have failed to replicate.  But what does failure to replicate mean?

Maybe it means that there was something wrong with the original study.   Maybe it means that there was something wrong with the replication.  But those aren’t the only options.  As nobel laureate psychologist Daniel Kahnamen wrote in an open letter to the scientific community:

In the myth of perfect science, the method section of a research report always includes enough detail to permit a direct replication. Unfortunately, this seemingly reasonable demand is rarely satisfied in psychology, because behavior is easily affected by seemingly irrelevant factors.

Note that underspecification of methods is an issue in all sciences.  Psychology just has a particularly rough time of it because psychology itself, like other soft sciences, is so underspecified.  Behavior is affected by seemingly irrelevant factors which are actually relevant previously unspecified factors.

In a better world, replication would be a collegial and common process involving many back-and-forths between originators and replicators.  Each replication could help identify new factors that turn out to be surprisingly relevant.  Eventually the hypothesis and methodology would be specified enough to permit consistent replication, at which point we’d have both our question and our answer.

This example makes clear that asking and answering questions are not two separate activities.  They are intertwined, at least when the questions and answers are new.  So it makes no sense to say, “Asking questions is harder than answering them” or vice versa, because you can’t do one meaningfully without also doing the other.

FYI: to read more about replication, try this article I wrote back in 2014 on the Open Science Collaboration blog: What we talk about when we talk about replication.

 

Hard and soft sciences

Back when I was a research scientist, I straddled the boundary between “hard” and “soft” sciences.  I did social psychology, which is a pretty soft science as sciences go, but I paired it with biology and physiology in general and endocrinology in particular, which meant getting a taste for some of the harder stuff.

I have never particularly liked the terms “hard” and “soft”, though, because it’s too easy to conflate them with “hard” and “easy”.  There’s a saying that goes: the soft sciences are easy to do poorly and hard to do well.  They are easier to do poorly than the hard sciences, and harder to do well than the hard sciences.  Here, have a chart:

What’s going on here?  The hard sciences are better developed than the soft sciences, so it’s clearer when someone’s making obvious mistakes, cutting corners, or making under-supported claims.  That makes it difficult to do poor work.  It’s also difficult to good work, of course.  The easiest thing to do in the hard sciences is to meet a minimum level of competency and do solid but uninspiring work.

Meanwhile in the soft sciences there’s questions even about the field’s basics.  There’s still a minimum level of competency, but it’s much less stringent than in the hard sciences.  So sloppy researchers tend to end up in the soft sciences.

Here’s another way to approach the hard/soft distinction. What’s easier, formulating questions or answering them?  It’s almost always easier to do the latter, provided you’ve very clearly and specifically formulated your question.  Of course we seldom do get our questions right on the first try, and so answering them involves going back over and over again and updating our hypotheses, which makes answering questions feel hard.  But the hardest parts of answering questions are really secretly still about asking them.

In the hard sciences, it’s easier to clearly and specifically formulate questions because so much knowledge has already been established.  Isaac Newton famously said (paraphrased) ‘If I have seen further it is only by standing on the shoulders of giants.’  The hard sciences are full of giants, with shoulders for modern researchers to stand on.  The soft sciences are by and large still on the ground.

For this reason, I prefer the terms “developed” vs “undeveloped” sciences.  I think it comes closer to the essential difference.

Note: this post has an update/correction post.