The peer review system no longer works to guarantee academic rigour

[–] [email protected] 15 points 23 hours ago

Honestly, reducing the teaching + publish-or-perish + the constant need to apply for grants would go a long way towards fixing the review process. Academics have to spend a lot of time doing a lot of non-academic work that peer reviewing properly sometimes gets pushed down the list of priorities.

[–] [email protected] 15 points 1 day ago

what we really need is funding for experiment duplication.

[–] [email protected] 35 points 1 day ago* (last edited 1 day ago) (2 children)

Maybe, just maybe, if editors did a hint of work with all the money they steal from public science funding, we could stabilise the system towards more integrity and less quantity of publication. Or also just get rid of editors to obtain the same result, but this is sadly utopic today. Peer reviewing is not the problem, and probably still is the best way to assess research quality. However, tendency towards quantity over quality, and applied research over fundamental are what skews the process and its results

[–] [email protected] 25 points 1 day ago (1 children)

if editors did a hint of work with all the money

Exactly. Why do authors need to pay for review/publication but the reviewers are volunteer and the journals paywalled? There is a fundamental mismatch between who gets vs deserves the money.

[–] [email protected] 10 points 1 day ago (1 children)

Wait, reviewers are not paid?!

[–] [email protected] 14 points 1 day ago (1 children)

Not typically, no. There have been exceptions ofc, but peer reviewing is typically volunteer.

[–] [email protected] 6 points 1 day ago (1 children)

I had no idea. That's fucked up.

[–] [email protected] 3 points 1 day ago (2 children)

Eh, it's not as bad as it sounds TBH. Paid reviewers would have ethical and economic pressures that hinder their effectiveness. Non-specialists in the same field would end up responsible for reviews of articles they are only rudimentarily familiar with (think astrophycisists working on exoplanet formation and composition having to review papers on black hole implications for dark energy. They 'could' but are not the best qualified to do so). Needing to review enough papers to earn a living means this dilution multiplied 100-fold to get enough done.

With volunteering at least scientists that are interested in that paper's topic, and hence are likely a specialist in it, are the ones looking at it and doing so at their leisure instead of needing to do 100 by weeks end to put food on their table.

Personally, I think all privatization involved should be removed. Volunteer reviewers to public non-profit journals paid either by donations or tax dollars and freely accessible to all.

[–] rockSlayer 2 points 23 hours ago* (last edited 23 hours ago) (1 children)

Why does the selection process for reviewers need to also disappear to provide wages? Journals still select reviewers for best fit. Ultimately, peer reviewers are performing labor for a corporation making profit from that labor. It is unethical for anyone to be put in a position to provide free labor in the pursuit of profit for a corporation.

[–] [email protected] 2 points 14 hours ago (1 children)

I'm arguing that the solution to your concern is that the corporation and profit aspects be removed. The reason I prefer this angle is because science emphasizes the need to remove all or as much bias as possible and economic incentives inevitably induce bias' and restrictions that increase the problems I've already pointed out earlier.

The question of whether peer reviewers ought to be paid and how is a complex one that has many ethical considerations on either side of the argument. I strongly recommend you research this debate yourself if you are interested in the subject.

[–] rockSlayer 1 points 14 hours ago

I agree that ultimately, science and profit do not mix in any capacity and the money aspect must be done away with. I do have some knowledge of the debate as a labor activist, but not nearly as much as I'd like. However, until there is an shift to economic socialism on an international scale, anyone doing science is performing labor that will produce an incentive to extract profit from these workers. Ideals and ethics are important considerations for science, but the class dynamic cannot be ignored and must be addressed for an equitable solution to emerge.

[–] [email protected] 2 points 1 day ago

Right, your take sounds convincing. Thanks for the insight!

[–] PriorityMotif 2 points 21 hours ago

The hell aren't colleges publishing this stuff themselves? There isn't an academic journal published by even one single university?

[–] spankmonkey 21 points 1 day ago* (last edited 1 day ago) (1 children)

There is literally no other option than peer review for science.

Does peer review need to work the way that it does now with publishers as gatekeepers and an expectation that work will be reviewed for free? No, the process should absolutely change but it will still require peers to review new papers. Publishing before review and anonymous reviewers is a terrible idea.

[–] Vorticity 22 points 1 day ago (2 children)

If you read the article, they are suggesting a different approach to peer review, not doing away with it. They want to find ways to build in incentives for reviewers to make it worth their while to review rather than allowing it to continue as something that scientists do out of a sense of obligation.

They have an interesting approach but I think it doesn't go far enough.

[–] [email protected] 4 points 1 day ago* (last edited 1 day ago) (2 children)

Couldn't you have researchers who specialize in finding "bugs" in published papers (yes, researchers already do this to each other), like we have QA testers or bounties for finding exploits? Is this too aggressive an approach for science? Should work for hard sciences, though.

[–] [email protected] 2 points 21 hours ago

i heard about a woman a while back that did exactly that: she read papers across disciplines and found doctored results etc… she’d found something like 10 papers that had fabricated data

[–] [email protected] 4 points 1 day ago* (last edited 1 day ago)

This is a great idea I think, but part of the problem with science versus programming is that they're just very different social environments, so the expectations, norms, and demands on each are very different.

Dependent down a little bit more, most research is done by people with phds or other advanced degrees (or pursuing them) in an academic job, and one of the conditions of attaining or maintaining that job is publishing. And these are the same people doing the peer reviews.

I think what this creates, even aside from the overwhelming volume and complexity of work, is a certain amount of grace amongst academics. That is, I think a fair number of peer reviewers are not only failing to rigorously grapple with the material that they review, but because of the small social mileux and shared incentives, they are incentivized to not be very rigorous in many cases.

Not saying peer review is without value, but how harshly would you want to challenge or critique the work of someone whom you may work alongside or under in the future?

[–] spankmonkey 3 points 1 day ago

I had a thought and didn't make it clear, added the last sentence that I'm referring to the new system allowing for anonymous reviews. That combined with publish before review is making a new system catered to malicious business interests. Tobacco companies would just love this system.

Now the idea of making the whole process more visible to a wider audience? Yeah, that could be a benefit.

[–] rockSlayer 14 points 1 day ago

Like paying them?

[+] bitchkat -6 points 1 day ago (1 children)

How about Ai?

[–] [email protected] 4 points 23 hours ago (1 children)

You.are joking, right? I'm just missing that implied /s?

[–] bitchkat 1 points 22 hours ago

If my company can buy an AI powered proposal manager, someone is working on AI peer reviews. So while I was joking, i'm afraid that its coming.

[–] [email protected] 0 points 1 day ago

Spinscore

Science