Researchers are spending more and more time writing computer software to model biological structures, simulate the early evolution of the Universe and analyse past climate data, among other topics. But programming experts have little faith that most scientists are up to the task.
Not surprisingly, scientists who are not doing science prove to be incompetent. Also, not surprisingly, this mostly affects “scientists” in certain fields where they are not concerned with doing science.
As a general rule, researchers do not test or document their programs rigorously, and they rarely release their codes, making it almost impossible to reproduce and verify published results generated by scientific software, say computer scientists. At best, poorly written programs cause researchers such as Harry to waste valuable time and energy. But the coding problems can sometimes cause substantial harm, and have forced some scientists to retract papers.
I’m surprised that anyone checked their results, considering how much most people rely on computer simulations to interpret reality.
One such error tripped up a structural-biology group led by Geoffrey Chang of the Scripps Research Institute in La Jolla, California. In 2006, the team realized that a computer program supplied by another lab had flipped a minus sign, which in turn reversed two columns of input data, causing protein crystal structures that the group had derived to be inverted. Chang says that the other lab provided the code with the best intentions, and “you just trust the code to do the right job”. His group was forced to retract five papers published in Science, the Journal of Molecular Biology and Proceedings of the National Academy of Sciences, and now triple checks everything, he says.
The basic problem is not the use of computer simulations as such. The problem is the assumption that a computer simulation of reality is the same as reality. This is equivalent to the problem that students in many fields have when they confuse reality with their dialectic about reality. It may be a beautiful structure, but if the results don’t correspond to anything in real life, it doesn’t matter.
Diane Kelly, a computer scientist at the Royal Military College of Canada in Kingston, Ontario, says the problem is that scientists rely on “validation testing” — looking to see whether the answer that the code produces roughly matches what the scientists expect — and this can miss important errors.
Hah! This is hilarious!
Many programmers in industry are also trained to annotate their code clearly, so that others can understand its function and easily build on it. But scientists often lack these communication and documentation skills.
Evidently, they are so interested in finding the evidence they expect for the truth they already know, that they are not concerned whether anyone else will be able to understand their work or reproduce it.
Aaron Darling, a computational biologist at the University of California, Davis, unwittingly caused such a mistake with his own computer code for comparing genomes to reconstruct evolutionary relationships. He had designed the program to work only with closely related organisms, but discovered that an independent group had used it to look at sequences far outside the code’s working range.
Again, this is mainly a problem for people who don’t study the observable physical universe in the present day; that is, people who are not doing scientific work.
Proponents of openness argue that researchers seeking to replicate published results need access to the original software, but others say that more transparency may not help much. Martin Rees, president of the Royal Society in London, says it would be too much to ask reviewers to check code line by line. And in his own field of astrophysics, results can really be trusted only in cases in which a number of different groups have written independent codes to perform the same task and found similar results.
More problems in a field characterized by naturalistic-sounding speculative metaphysics.
This would be of absolutely no concern to me if there were not so many morons walking around citing “scientific truth” to justify their preposterous assumptions about the world. Then it gets worse when people start trying to figure out new ways to manipulate the gullible into accepting more domination by the political animals.