Science fiction? Why the long-cherished peer-review system is under attack

Tom SPEARS

Mathematicians have been studying the number pi for thousands of years, so it might seem startling to learn that a gentleman in Athens, Wisconsin suddenly changed its value.

His revised pi is a bit bigger than the one everyone else uses. And it stops after 12 digits instead of running on forever.

To any trained mathematician, this isn’t even worth a second look. It’s the work of an amateur who doesn’t understand pi, the ratio of the circumference of a circle to the diameter.

But his paper, published in a fake science journal that will print anything for a fee, now shows up in Google Scholar, including footnotes citing himself, himself, himself and himself again. Oh, and Pythagoras — once.

Google Scholar is a search engine that looks for scientific articles and theses, the meat and potatoes of scientific literature.

But Google Scholar is not discerning. It also turns up a new paper from an Egyptian engineer who decided to rewrite Einstein and claims to have discovered the nature of dark energy at the same time.

Again, that’s an eye-roller for anyone in the physics business, yet there it is in a search of scholarly journals, muddying up the intellectual waters.

It wasn’t supposed to happen this way, of course.

The peer review system was designed to ensure that before research is published, it’s of good quality, whether everyone agrees with its conclusions or not.

Under the system, a researcher who makes a discovery sends it to a science journal to publish. The journal sends it to a group of experts in the field to check it out to see whether the work is well done. If the peers approve, it is published — often with changes requested by these experts.

But peer review is under assault, from both the outside and the inside.

Thousands of “predatory” publishers that imitate science journals are undermining scientists’ ability to distinguish good from bad.

The predators skip peer review. A series of tests by the Citizen, Science magazine and others found that many predators will print anything verbatim, allowing a flood of low-quality work to appear in online journals.

Researchers in Canada often maintain they are skilled enough to recognize and ignore worms in the scientific apple. But there’s a harder question: if simply godawful papers are getting published, what about the whole blurry spectrum ranging from substandard through so-so to slightly plagiarized?

It’s not just an academic question. Our daily use of technology, and even the medical treatments we receive, depend on the ability of researchers to trade information honestly.

Many academics see the rise of a two-tier science system, a stronger one found in developed countries, and researchers with fewer resources in the developing world who depend on predators. A Turkish newspaper recently wrote that whoever pays the predators “climbs the career steps two by two.”

There are also small open-access journals struggling to set themselves up with insufficient resources. One in Ottawa claims to publish 23 international journals, using freelancers, from a single room in Billings Bridge. Another operates from a house in Burnaby, B.C. Staff who work there won’t reveal their last names.

Scam scientific conferences are also a growing assault on the peer review process. These events allow anyone to register, for a fee, to present a paper on anything, good or bad, on topic or off.

The speaker can then go to the dean of his or her faculty and say, “Look, I’m the keynote speaker at a conference in Paris!” It seems like a career-boosting move, especially if the name of the conference mimics that of a real scientific gathering.

It’s all about blurring.

WASET, the World Association of Science, Engineering and Technology, organizes conferences somewhere in the world every few days, but they’re low-quality affairs at which anyone can register a paper on anything.

To dress it up, WASET offers conferences with names the same or similar to real conferences organized by real scientific groups. Recently, WASET put on an International Conference on Educational Data Mining. The real version belongs to the International Educational Data Mining Society.

In October alone, the Turkey-based WASET has set up conferences in Bali, Brussels, Osaka, London, Paris, Dubai, Barcelona and Istanbul. Each will cover dozens of fields — aviation, agriculture, business management, linguistics, mathematics, pedagogy, biology, law, medicine, computer engineering, nanoscience, history, civil engineering, geology, chemistry, ecology and on and on.

Anyone who presents a paper pays 500 euros, roughly $700, (and 100 euros more if they want the paper published.)

It is tough to shut down groups like WASET. It is not against the law to stage a conference with the same name as another, and even if authorities were to go after such operators, they are hard to identify and could easily pop up under a new name a week later. WASET has scheduled 103 conferences for next year, mostly in Western Europe and two in Canada. In most cases, universities pay for the travel.
Canadian academics like to say that we don’t get fooled by scams. But at the University of King’s College in Halifax, science historian Gordon McOuat noted that his university has had to cancel travel plans of faculty wanting to attend scam conferences.

Jeffrey Beall of the University of Colorado found a new publisher of 107 online journals — sprintjournals.com — that advertises an affiliation with the established Elsevier publishing group. There’s no actual connection, but anyone reading Sprint’s website will see the familiar Elsevier logo, and may trust it.

Real journals are “hijacked” by impostors using the same name. So an article published in Afinidad can be either in a reputable journal from Spain or an impostor using Afinidad’s name to trick authors. This is a widespread practice. An Indian publisher has a website mimicking the legitimate BioMed Central.

The fake journal Experimental & Clinical Cardiology used to be real until it was sold and new owners took a more profitable path. It blurs its identity by using the same name as before and by claiming to be the official journal of a very real and legitimate cardiology society, although there is no connection.

Legitimate peer review is far from dead. But it has a nasty cough that isn’t clearing up.

The papers on pi and dark energy are just two of many that show up in academic index services.

“For example, Google Scholar does not screen for quality, and it indexes many articles that contain pseudo-science in them,” Beall says.

“(It) is the world’s most popular index for scholarly content. This index and many other abstracting and indexing services do not sufficiently screen for quality and allow much scientific junk to be included in their databases. This affects the cumulative nature of science, where new research builds on the research already recorded in the academic record.”

And the predators have other ways to gain acceptance.

“Some scholarly publishing organizations do not screen applicants for membership. Thus some predatory publishers apply and are granted membership. Then the predatory publishers use these memberships to argue that they are legitimate publishers.”

Even the top ranks of peer review have their problems, though.

Two U.S. psychologists ran a test of peer review back in 1982, taking 12 papers that had been published in high-ranking psychology journals and re-submitting them to the same journals with changed titles and different authors’ names.

All the originals came from famous institutions. The re-submitted ones carried names of fictional authors and institutions, some on the hippy-dippy side: the “Tri-Valley Center for Human Potential,” for instance.

The same body of work should be accepted again, the two researchers felt. It wasn’t. Only three of 38 reviewers and editors spotted the duplicates. That allowed nine papers to continue, and eight were turned down for allegedly poor quality — evidence of bias, the authors concluded, against academic institutions with lower pedigree.

(Giving preference to well-known academics is known as the Matthew Effect, from a passage in the Gospel of Matthew: “For to everyone who has, more shall be given…”) The journal Science flagged it as a problem as far back as 1968.

Yes, but people are smarter now, right?

Not so fast. In 2006, the editor of the British Medical Journal (BMJ), Richard Smith, listed problems with peer review. It’s inconsistent, he found: two reviewers of the same paper can come to “laughably” opposite conclusions. Sometimes it’s dishonest (as when a reviewer rejected a paper, but stole chunks of it for his own work.) It rarely catches fraud. And it’s tilted in favour of male researchers.

Smith, who was also chief executive of the BMJ Publishing Group, calls peer review “little better than tossing a coin” and “a flawed process, full of easily identified defects with little evidence that it works.”

And he cites a colleague with similar misgiving: “That is why Robbie Fox, the great 20th century editor of the Lancet, who was no admirer of peer review, wondered whether anybody would notice if he were to swap the piles marked ‘publish’ and ‘reject.'”

Its nature, he writes, is impossible to define precisely. “Peer review is thus like poetry, love, or justice.” (A side note: Smith rented a 15th-century palazzo in Venice to write this as part of a longer analysis, and he clearly had the arts on his mind, comparing a medical study at one stage with an altarpiece by Tintoretto.)

Still, Smith notes, the scientific establishment believes in peer review, and concludes wryly: “How odd that science should be rooted in belief.”

This summer the British Medical Journal published a study of how well peer review worked in 93 recent medical trials.

It explained: “Despite the widespread use of peer review little is known about its impact on the quality of reporting of published research articles.”

It concluded that “peer reviewers often fail to detect important deficiencies in the reporting of the methods and results of randomized trials,” and they “requested relatively few changes for reporting of trial methods and results.” Most of their suggestions were helpful but a few were not, it added.

Sometimes, even with the names removed, reviewers may recognize an author’s research because everyone works in the same field. Christine Wenneras and Agnes Wold, two Swedes analyzing peer review in Nature in 1997, wrote of the ”friendship bonus” and “nepotism” that can occur.

As a widely quoted 2006 opinion piece in Nature by Charles G. Jennings, one of the journal’s former editors, noted: “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.” Jennings went on to argue for harder, more quantifiable factors to make peer review more dependable — work currently under way by an international group called EQUATOR which promotes strict reporting guidelines.

Medical professor Roger Pierson of the University of Saskatchewan points to the flip side: rivalry.

“Human nature being what it is, professional jealousy and egos flare from time to time just because people don’t like each other — they’ll trash each others’ manuscripts to be spiteful,” he wrote in an email.

“There are also constant cases of people in races to claim ‘First!!!’ for whatever that’s worth,” and giving a negative or delayed review to one’s rival “can give them an advantage,” he wrote. “The science community and the university system have no real way to respond or ensure that their members are playing nicely with one another. C’est la vie.”

These days, some scientists skip the whole traditional publishing process, at least for some of their work. The Internet beckons, and they go straight to their audience, cutting out the middleman.

This is what University of Ottawa biologist Jules Blais calls “the blogification of science.” It doesn’t replace traditional journal publishing, but “this is something that we have been seeing with social media. The volume has gone way up and the quality is coming down. We have to be very careful in how we preserve our highly regarded peer-reviewed publications because we need them desperately.”

Another way to bypass peer review is to post work directly online at arXiv (pronounced “archive,”) hosted by Cornell University. It takes papers in mathematics and some sciences, including physics and astronomy. The system is called “preprint,” implying that papers can go online at arXiv while awaiting peer review somewhere else. But the second stage isn’t mandatory, and there are now more than 8,000 papers a month posted on arXiv.

Still, they can’t ignore the traditional journals entirely. Careers are built there. Nature estimates academics worldwide publish more than one million papers a year.

“Everything we do is really judged on publications, and if we want grant funding (to keep a lab running), people look at your CV,” says Joyce Wilson, a virus researcher and relatively new associate professor at the University of Saskatchewan. “And if you have published well in the past they assume that you will publish well in the future, and they will give money.”

All of this worries Saskatchewan’s Pierson. “Peer review doesn’t catch many things, even patent fraud,” he says. “There is a growing literature on this one. Fraud and deceit in the halls of science have been around forever and now that careerism seems to have become more important than the search for truth that many, if not most, of us actually entered the biz to pursue…. well, things have progressed.
“Even the biggest, most prestigious journals are not immune. There have been some particularly egregious cases in the past decade. So, cutting the garbage? Perhaps not as much as we would like to believe.”

Still, he concludes, that while peer review is far from perfect, “right now it’s the best we’ve got. I think that the system needs an overhaul and perhaps this issue (bogus science) is a good stimulus.”