4
Breaking Down Cass Review Myths and Misconceptions: What You Need to Know – The Quackometer
(www.quackometer.net)
this post was submitted on 14 Apr 2024
4 points (62.5% liked)
Skeptic
1332 readers
1 users here now
A community for Scientific Skepticism:
Scientific skepticism or rational skepticism, sometimes referred to as skeptical inquiry, is a position in which one questions the veracity of claims lacking empirical evidence.
Do not confuse this with General Skepticism, Philosophical Skepticism, or Denialism.
Things we like:
- Civility
- Thoughtful discussion based on evidence and facts
- Humor
Things we don't like:
- Personal attacks or disrespectful attitude
- Wild speculation on events with no evidence
- Low-effort comments and posts
Other communities of interest:
"A wise man proportions his belief to the evidence." -David Hume
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
This is the lie. They didn't dismiss all but two studies, they actually included 60. More than half of the 103 studies identified for the review.
So yes, Erin, and now yourself, are peddling a lie.
It's key part of synthesising multiple sources into a meta-analysis. Including poor quality studies dilutes the quality of the overall analysis.
By design, it's biased towards higher quality research.
Synthesis is a paragraph summary inclusion ONLY, it means they didn't use data from the study, it is dismissal. I'm done arguing that with you.
They have absolutely used the data from those 60 studies. You can read where they say explicitly that in the report if you cared to.
You are utterly mistaken and firm on your conviction, these are not the qualities of skepticism.
"Don't seek refuge in the false security of concensus"
That's not what synthesis means. I've written synthesis reports before and the data you include from those reports once you have dismissed them as inaccurate, it is an entirely selective process of whatever you want to include from them. We even have a phrase for it in law, Summarily dismissed.
And of the 103 reviewed they included data from 60. It is a lie to say they "dismissed all but two."
Legalese is irrelevant. A systematic review of scientific literature is a different beast to "writing a few synthesis reports".
So you don't know what you are talking about. Gotcha.
Synthesis reports in a scientific study when presenting data, are the parts of the report where you explain why you are dismissing data, so in this case ~98% of the data or studies. So what you just said is ~98% of the data was included in the synthesis report. that's not inclusion of the data. That's selective inclusion to support a conclusion. A normal scientific study can't dismiss 98% of available data. That reveals bias.
You can read the reviews for yourself
https://t.co/82Rjs2L1pA
https://adc.bmj.com/content/early/2024/04/09/archdischild-2023-326669
Let me know where you find the bit where they dismiss 101 out of 103 papers.
Hint, they didn't. It is lying to say they dismissed 98% of the studies they looked at.
Read it. Their only inclusion in the report is to half explain why the were discluded, exactly what I said. Most of the dismissals are unscientific, not supported by a statistical analysis of why it was discluded. Data doesn't become unreliable just because it is incomplete.
That report is absolutely rife with white washing and selection bias, I'd expect a scientific review of trans literature and studies to be a book at this point not 32 pages dismissing 98% of the data. It's frankly insulting to anyone that's read or written any number of scientific studies.
They did not dismiss 98% of the data.
Putting 98% of the relevant available data in a supplementary table like 4 is not including the data.
Supplementary Table 4 (from the first review) is a list of each of the 53 studies included in the review and how they were scored based on the Newcastle-Ottawa scale.
The "data" is in supplementary tables 3, 5, 6 and 7. Only studies that were scored as low quality were excluded from the synthesis.
"They dismissed 98% of data" is a lie.
No it's not. None of the dismissals are statistically/ scientifically supported, and the data they present is blurbed and incompletely presented in a way that isn't inclusive of what those studies actually say.
Nothing was dismissed at all (and "statistics" has nothing to do with it so curious to mention it).
Studies were scored for quality on the well established Newcastle-Ottawa Score. High and Moderate quality studies were included in the synthesis. Low quality studies were not, but their outcomes are still reported.
Outcomes from each study were included in tables 3, 5, 6 and 7.
'They dismissed 98% of the data" remains a lie.
You can't remove a study from a scientific paper without having statistical analysis to back it up. Each of those removed studies all had a statistical analysis of how confident they remained in their data even with the gaps. Because there aren't completed 100% studies in science it just doesn't happen so you use the data you have and test it for a confidence value you obtain using statistics. And the idea that some trans people don't make it to the completion of a study due to personal reasons or even suicide isn't that rare. Not using 98% of the data because of that would be stupid.
You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality.
Studies are also scored low on quality if, for example, they don't control for important sociodemographic confounders. Study that do control these, will have more reliable results.
You can read how the scoring works in supplementary material 1.
"They dismissed 98% of the data" remains a lie. Repeating it doesn't change anything.
"You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality".
That's wrong when you are trying to be scientifically correct. A science paper without that math isn't science my dude. And comparing trans healthcare data to Facebook polls is ridiculous
It's remarkably common in systematic reviews, a feature even. You give the impression that this is a new or foreign concept to yourself and are just encountering these ideas for the first time.
Search on pubmed or the bmj or the Cochrane library for other systematic reviews using the Newcastle-Ottawa score. You'll trip over them.
One of the studies reviewed recruited patients over Facebook and polled them.
"They dismissed 98% of the data" remains a lie.
Again I've written these reports. It is absolutely not common practice to disclude data without scientific reason and analysis. It is explicitly taught not to do it that way in college. And it is not scientific to do that without a statistical threshold and confidence analysis of your reasoning.
I am forced to strongly doubt this given your whole misunderstanding of the basic concepts on assessing methodical quality..
Certainly, you've never authored a systematic review for a reputable medical journal.
But don't take my word for it...
https://handbook-5-1.cochrane.org/chapter_13/13_5_2_3_tools_for_assessing_methodological_quality_or_risk_of.htm
You mean such as using a method like the Newcastle-Ottawa score to assess data quality?
If your college course covered systematic reviews and didn't include a review of study assessment methods, ask for a refund.
Statistics are not required to assess that a study without a comparator is weaker than one with.
"They dismissed 98% of data" remains a lie.
The Newcastle method is not seen as a scientific basis for dismissal on its own.
98% of the data was dismissed in the synthesis and was not used to reach the conclusion that there wasn't enough scientific evidence to support transition when 98% of the science says that is wrong.
And every scientific paper is expected to be comprehensive on its subject matter and/or thesis.
It's not used for "dismissal" it's used to score studies on their likelihood of bias. Studies without appropriate controls for example are more susceptible to bias than those with.
Demonstrably false, only low quality studies were excluded from the synthesis which account for less than half of the 103 reviewed. A lie is a lie no matter how often repeated.
That's not what the conclusions say, for example:
And
"They dismissed 98% of data" remains a lie.
https://www.tandfonline.com/doi/full/10.1080/26895269.2024.2328249
That was published a month before Cass came out and so hasn't anything to do with the two systematic reviews being discussed above. It doesn't even mention them.
I'm uncertain what expertise a business graduate can bring to assessing the quality of a systematic review in medicine.
Readers are free to Google the author and subsequently make a judgement on their objectivity on the subject matter.
And yet you have no scientific reason other than an ad hominem fallacy with the author with which to dismiss the criticism with. That like the Cass report are not scientifically sufficient reasons to disclude the criticism or the data respectively.
And I can garuntee you that the Cass report was not peer reviewed like all of the studies they dismissed were because it would have been torn apart. That's the real litmus test of scientific debate.
If they made a scientific argument about these review papers under discussion I might but this is just a polemic using unscientific language like "cis-supremacy" in a low impact obscure journal.
Newcastle-Ottawa scoring is a scientific method for weighting the methodical quality of scientific studies.
It was peer reviewed since thats BMJ policy, unless you have evidence to the contrary. There is even a link on the online edition of both reviews for you to submit a rapid response pointing out all their flaws which I would encourage you to do.
Interestingly some nice fellow DM'd me with a link to "Patient Zero" of the "they dismissed 98% of the data" myth.
https://twitter.com/benryanwriter/status/1779671152148857212
And of course, everyone has doubled down rather than admit they read the wrong paper. A better "litmus test" of scientific debate is humbly correcting yourself when shown to be wrong.
"They dismissed 98% of the data" remains a lie.
98% of the data could be summarized in one sentence. Trans healthcare and hrt works. 98% of the data comes to that conclusion with vast consensus across multi disciplines and fields comes to that conclusion and that was ignored. 98% of the data was discarded. Most of those studies discarded already had a statistical analysis backing up their efficacy while the Cass report doesn't. Nor does the Cass report include a nearly mandatory implicit bias report.
Those peer reviews are most likely selected and not randomized selections or contestations as most peer reviews are required to be, they are ok for initial release irc. But it is an outgoing process that doesn't have an endpoint. They were most likely provided prior to release and the normal peer review process won't be completed for years to undo the damage. But it is not considered peer reviewed yet.
Again you have not proven that the new castle Ottawa scale has any efficacy or scientific merit as a disqualifying tool No one has as far as I know.
Liar.
So now they were peer reviewed but by people you don't trust based on the same evidence you used to assert it wasn't peer reviewed in the first place I.e. zero.
You'd better tell the Cohcrane library to bin every systematic review they've ever done which used this system then. I'd be eager to hear their reply to you
98% of the data was discarded
No I'm just explaining the process and why it isn't complete yet. Or even valid yet
And show me that the Cochrane library ever discarded a study using the criteria even once yet alone with the same level as the Cass report and I'll write them
For something that illustrates the problem with the Cass report read https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/
It was not. All studies that scored high or moderate quality made it into the synthesis. That's 60 out of the 103 looked at, that's not 2%.
You are speculating, based on nothing.
Here's one I found in <7 seconds
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013256.pub2/full
If you want to find more simply search the Cochrane library for reviews with "Newcastle-Ottawa" in the main body of text. It seems like this is new to you.
The relevance of a joke paper from 2003 to a systematic review published last week is certainly questionable but it seeks you're trying to imply that Cass discarded anything except RCT's. The didn't and that''s myth #2 from the original Quackometer article.
Where will the goalposts move next?
Goal posts haven't moved and I've already pointed out a dozen of so methodological flaws around the Cass report that you are choosing to ignore.
That's on you
You haven't pointed out, let alone substantiated, any. If you truly believe you have then I implore to use the rapid response function on the bmj site and communicate these catastrophic flaws to the editorial team immediately. I'd be eager to know what their reply is.
I don't need to, it is already happening within the scientific community of which I am merely a part.
Don't abdicate responsibility to someone else, you've clearly got a firmer grasp of the issue than the editorial board of the British Medical Journal. You would be neglecting your duty as "part of the scientific community" to abdicate responsibility on such an important matter.
Indeed the whole medical establishment must be told about the critical flaws in the Newcastle-Ottawa scoring system before other medical scandals are allowed to happen. Imagine having that on your conscience.
Lol first sign that you might actually be human.
And it has already been widely criticized before that's why there was the parachute joke report. Hence it is already the brunt of jokes to use that scoring scale.
So strange that everyone waited over 20 years and 100's of systematic reviews in medicine and science before, serendipitously, discovering that the Newcastle-Ottawa Scale was infact no good during these two particular reviews into trans care in the UK.
Just what are the odds?
No the Cass report is just misusing the scale. It's not a disqualifying tool and the scale still has uses which just means further analysis into the subject matter. Which is why the Cass report needed to be books longer, it's not comprehensive.
That's a new goalpost. It's being used by Cass exactly the way it's supposed to by scoring studies based on their susceptibility to bias.
If you'd bother to read that similar systematic review on postoperative inflammatory bowel disease you would have seen the exact same usage.
Again that's a joke to do that.
Damn here's another "joke" about contraceptives and bone fractures
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009849.pub3/full
And one another abput yellow fever and HIV
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD010929.pub2/full
And influenza vaccines in cancer patients
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD008983.pub3/full
And there's another 96 on the first search tab alone!
Just what are those clowns at the Cochrane Library up to eh?
Relevance?
They all use the same Newcastle-Ottawa system to score studies based on their likelihood of bias in the exact same way the Cass reviews do. The method you described as a joke.
It's not an indicator of bias, no causal study has been done to show that there is a relationship between bias and the Newcastle Ottawa scale
Studies that self select their cohort and don't include adequate controls are more susceptible to bias than those that do otherwise. Evaluating studies based on their susceptibility to bias is a vital part of the systematic review process.
You can read more about it here https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
But not actually proof of bias.
Ah young padawan, there is no such thing as proof of bias. There is merely the risk of susceptibility of it.
Exactly which is why the Ottawa whatever standard is not sufficient to discard a study. You have to do more.
Again, you really need to feed this startling discovery back to the medical community which has been using NOS for over 20 years. What a scandal.
No the medical community largely respects the short comings and uses of the Ottawa protocol. That's what made Class's report so insulting.
Blimey, you're speaking for the medical community itself as a whole now, pray tell then why haven't they binned every systematic review ever carried out using the NOS system?
Why after 20 years of use is this system only being rubbished after two reviews into gender affirming healthcare in the UK were published?
Why are you the only person complaining about the Newcastle-Ottawa system when everyone else online is making up lies like "98% of data was dismissed"?