I ate dinner last night at Cuban Revolution, a local restaurant (trust me, this will eventually be about science). They put a heavy emphasis on presentation: when you walk in the restaurant, you are greeted by dim lighting and a string of blue lights on pillars; the menu itself is disorganized, with text in five fonts and five colors; they insist each and every one of their sandwiches are award winning. This emphasis on presentation only lowered my expectations of the food. I ordered ropa vieja, which I believe is Cuban for Pad Thai, and was both relieved and disappointed by how adequate it was. Thinking about why, I realized that I probably could have cooked something similar myself.
I am a mediocre cook. I started by boiling pasta, but have moved up to complicated techniques like pan frying. My specialties include lasagna, various marinades, and kibbeh. When I got to restaurants, my first criterion is, it has to be better than what I can make myself. As I've improved my cooking skill, the quality of restaurants I am willing to pay for has increased. TGI Friday's or cheap Chinese stir fry no longer cut it.
I read science papers in much the same way I evaluate food: could I have done it myself? Could I have performed the experiments? Or if the experiments are simple, do I know enough math for the analysis? For papers with simple, tedious methodology, I end up empathizing with the authors, that they performed so many experiments. For technique papers, I am impressed the authors were able to get their technique to work, even if they don't answer an interesting biological question.
Like my criterion for restaurants, my expectations for papers have shifted as I have learned more techniques, and how difficult (or easy) they are. When I was young, my eyes glazed over every western blot, and I struggled to keep IPs and antibodies straight when interpreting blots. A few years ago, however, I tried Western blotting myself. It was, of course, a disaster: my bands all smiled, my total protein levels were inconsistent, and my activators never activated. I realized that every biochemist must troubleshoot all of these minor details to get consistent results. Now, my eyes still glaze over when I see figure upon unending figure of western blots, but it is the glaze of respect. I would compare these papers to lasagna, where you don't feel guilty ordering it because you know how time consuming and tricky it is to make.
On the other hand, there are technical papers which I respect for how much math is involved. For example, the concepts behind STORM or PALM imaging are quite intuitive, but the math and programming behind them may be beyond my reach. Reading these papers is like eating at a molecular gastronomy restaurant, where amazing wizardy went into creating the food, but you only care that it tastes good.
Orthogonal to the easy/hard axis are chemistry papers (or more likely figures for me), where I simply have no idea what's going on. Mass spec? Baked alaska? How'd they do that?
In the middle are the papers that use techniques I am familiar with, or even use every day, like electrophysiology or imaging. For these papers, I have the highest standard. Are their images clean? The statistics fair? Do the experiments address their questions? For example, in my first blog post, I was critical of a recent paper on PI3K and AMPAR trafficking, two topics I know comparatively well. These papers are like going to a restaurant and ordering the marinated chicken: it had better be damn good you want me to come back.
When scientists discuss papers amongst themselves, we usually emphasize whether the results are believable, or conclusions justified. Reading papers alone, however, the first question I usually ask is, can I cook it?
