"Jim -- Did you catch that show last night?
Pam -- No, I don't watch TV. I have a life.
Jim -- Really? What's that like?
Pam -- It's nice. You should get one.
Jim -- But then who will watch my television?
"
- the NBC sitcom "The Office"
Monday, November 30, 2009
Chicago Boyz write more on those climate-predicting computer models and the practicality of "peer review":
Thirty years ago, most scientific software was no more complicated than your average household spreadsheet is today. Software was mostly just a numerical summation tool that merely accelerated the processing of data. If a scientist had a computer, great, but if not it didn’t change the actual conclusions of their experiments. As a result of software’s relatively trivial nature, peer reviewers, journal editorial boards and other scientist paid little attention to the software that experimenters used to produce their results.I'm banging on this a bit because I'm kind of ashamed that I never thought of it. As I mentioned in a previous post, I've long been skeptical of computer-based climate models, mainly because I know what computers are good at and what they're bad at, and one thing they're bad at is predicting the future.
Unfortunately, that attitude has persisted even as software has grown from a minor accessory into the tool that actually performs the experiment. Today many papers are nothing but reports of what a unique piece of software spit out after processing this or that great glob of data. These software programs are so huge, so complex and so unique that no one who wasn’t directly involved in their creation could hope to understand them without months of study.
. . .
The practical inability of peer reviewers to verify scientific software doesn’t mean much in reality, because scientific institutions never even developed the standard that experimenters had to make the code for their software available to reviewers in the first place!
This raises a troubling question: When scientists tell the public that a scientific study that used a large, custom-written piece of software has been “peer reviewed” does that mean the study faced the same level of peer scrutiny as did a study that used more traditional hardware instruments and procedures?
Clearly not.
But I've never really put two and two together when it comes to the oft and confidently stated "AGW is settled science. Our findings are peer reviewed".
I never asked what was meant by "peer reviewed". Is that an appeal to authority that isn't as strong in reality as it sounds in rhetoric?
And just how good are those computer climate models? Seems that, at least in the example of the East Anglia CRU models . . . not so great.
Trackbacks:
Trackback URL: http://thinklings.org/bloo.trackback.php/5683.
Comments on "More on Climate Models and Peer Reviews":
2. FrancisT
- 12/01/2009 11:43 am CST
There is clearly a disconnect between the theory of the scientific method and that practised by numerous modern scientists - including climate scientists. This affects the whole "repeatability" thing. Scientists just don't seem to actually repeat other's experiments or calculations. Instead they take the end results and extend them.
If you read Feynman's "Cargo Cult Science" you see that the rot had started decades ago but we've reached a pretty poor point now in terms of scientists doing precisely the things that Feynman warns against (and not doing the things he says you must do).
When the only people who looked at the output critically were other scientists who behaved similarly this was no problem, unfortunately the internet and the poer of modern PCs means that anyoen can run the calculations of a particular paper they want to. And some try to and discover that the necessary data (and code) just isn't there.
3. Raindream
- 12/01/2009 1:19 pm CST
Someone on my blog noted that he'd like to see a government-funded study NOT conclude that more government-funded research is needed. This is so very ugly.
4. Bill
- 12/01/2009 5:19 pm CST
Bob, thanks for the insight.
I do agree with you on coding standards, believe it or not :-) - I don't take them as the problem per se, but rather as a symptom of a bigger problem. A carefully and well-coded program is less likely to have hidden bugs (note: even if the formula's right, it can produce wrong or inconsistent results with buggy, cr@ptastic code). Well-coded programs are less likely to have deep-buried constants that also exist in a database and, even when changed there, don't get reflected because a lazy coder hardcoded them "just temporarily, to test" and then forgot to remove the constant. They are less likely to be "accidentally" pointed at the test database rather than the production database, are less likely to have failures that go unnoticed because of bad error-reporting.
All that being said . . . I read your comment and defer both to your experience and to the argument you made. I'm with ya.
5. Bob Sacamento
- 12/02/2009 9:16 am CST
Bill,
Yeah, we are in 95% + agreement.
And despite my longwindedness on coding standards, the most important (relatively speaking!) point I had was about scientific codes becoming standardized. Unfortunately, that has really taken the fun out of the work. Oh well. I'm sure they said that about the printing press too.
Thanks.
Leave a Comment:
Having done alot of computational science myself (but not climatology) I come at this from sort of the opposite side of the coin. These guys are wrong about a couple of things, like
Thirty years ago, most scientific software was no more complicated than your average household spreadsheet is today. Software was mostly just a numerical summation tool that merely accelerated the processing of data.
Thirty years ago, sophisticated simulation programs were already very prevalent in the literature. Needless to say, they are moreso now, of course. But they were there, and important, even back then.
Today many papers are nothing but reports of what a unique piece of software spit out after processing this or that great glob of data.
Not in the best journals. Researchers who get "surprising" results are expected to explain why their results differ from the established norms. If they can't do it, they usually don't get published. I have seen this happen even to a guy who had results from a code that was probably very good.
These software programs are so huge, so complex and so unique that no one who wasn’t directly involved in their creation could hope to understand them without months of study.
This is true, and has been a big gripe of mine for a while now. But there are error-correction mechanisms in place. 1) This is going to tear the heart out of any computer scientist out there (sorry Bill) but in a community of smart people trying to do things right and checking on each others' work, formal coding standards just aren't as necessary as they are in a commerical environment. I say this as someone who has spent a good deal of time in both environments. I have seen scientific codes give wrong results because they used equations that were inappropriate to the regime being studied; I have seen codes give wrong results because the algorithm they used was inaccurate in the regime being studied. True. But these problems can be eliminated only by a knowledge of physics, chemistry, what have you, not by coding standards. I have never -- to my knowledge -- seen a code give a published wrong result just because somebody wrote the code wrong. But if that gives you heart burn, my most important points are 2) People have competing codes that give competing results (see above). The community of scientific coders, when acting correctly, acts as a fail-safe (see below). 3) As time has gone on, the big simulation codes are becoming standardized, even "commidified", so that less and less scientists are writing their own code, and they do have the resources to examine the code they use.
I said it before, but I'll say it again. The problem with the climatologists is not that they relied on big simulation codes to do their work. They really had no choice. The problem is that they ran their simulations, and then didn't wait for the real data to come in before they started screaming. And also, as I suspected, and has now become clear, the climatology community was not acting as the fail-safe they should have been. They had gone down the group think hole.