News

Yes I was, i just wasn't fooled.

Edit: gonna drop this great explanation here and call it good, I don't think I need to entertain mbfc skeptics any longer like they have some sort of valid point. It's mostly users who don't like the idea that they get their news from propaganda, or don't like that a highly reliable and factual source challenges their views... Like, for example, here.

https://skeptics.meta.stackexchange.com/questions/4589/is-media-bias-fact-check-a-reliable-site

I waited 2 weeks before posting this answer to give other's the opportunity to weigh in and do research. Unfortunately that hasn't happened. I didn't want to come off as "of course it's reliable" because of confirmation bias or falling into the appeal to authority fallacy. Since there isn't anyone else that wants to put their foot forward, I will, with lots of research.

TL;DR

Yes, Media Bias/Fact Check is normally reliable. It's not 100% reliable, but nothing is. It's research is done by humans who have biases, even if they are unconscious biases. However, when they are made aware of mistakes, they take steps to address the issue.

Baseline Research

In order to validate if Media Bias/Fact Check (MBFC) is reliable, I need to set some sort of baseline as to what's reliable, what's not reliable, and how I determine reliability. So I needed to research other sites I included in my research. Yes, we're going down the rabbit hole, Dorothy, but I'm cutting it a bit short so this Answer eventually ends.

I've left off research on sites that were clearly biased in their speech patterns. I've also avoided articles behind paywalls, mostly because I don't have access to those articles, either. I've gone through dozens (maybe even over a hundred or more) articles, sources, opinion pieces, and more to get as relevant and unbiased information as I can.

Snopes

After reseaching this site, I find it to be reliable. Not only based on authoritative/reliable sources, but also it's generally accepted by people in general to be reliable.

https://en.wikipedia.org/wiki/Snopes

https://www.bbc.com/news/world-us-canada-48867870

https://www.makeuseof.com/tag/is-snopes-biased-reliable/

https://credibilitycoalition.org/credcatalog/project/snopes/

https://www.jacksonville.com/article/20120928/NEWS/801246493

and many more......

Ad Fontes Media Bias Chart

I'd rate this also as reliable. It gets updated and has it's methodology published. The research is published, but seems limited in some cases. However, sites with hundred or tens of thousands of articles can't all be checked for biases. Maybe they will in the future, but that's unreasonable today. It's a fairly new chart and has it's critics, though.

https://bigthink.com/politics-current-affairs/media-bias-chart

https://www.cityam.com/how-to-read-financial-news-coronavirus-confirmation-bias-and-political-bias/

https://www.marketwatch.com/story/how-biased-is-your-news-source-you-probably-wont-agree-with-this-chart-2018-02-28

https://lifehacker.com/check-a-news-sources-bias-with-this-chart-1844330891

https://credibilitycoalition.org/credcatalog/project/the-media-bias-chart/

AllSides

AllSides also seems to be reliable, however this is a essentially a polling site, so it relies on people's personal opinions, rather than a professional researcher(s). It's also exclusively for the USA.

https://www.commonsensemedia.org/website-reviews/allsides

https://www.slant.co/options/32441/~allsides-review

https://www.ksfr.org/post/allsidescom-calls-out-media-bias-both-sides-promoting-democratic-dialogue-and-free-speech

Many sites (including MBFC) are listed and rated well in this PDF by Unesco:

https://en.ccunesco.ca/-/media/Files/Unesco/Resources/2019/08/AGuideForTheConscientiousCitizenChristopherDornan.pdf

I could go on, but this seems like it's getting redundant. Many of the articles that I've read include references to multiple fact checking sites, which generally include MBFC, which I've been saving for now, the research of MBFC.

By the way, the reason I did this "baseline research" is so that I could compare if MBFC is as reliable as these other fact checking sites that are considered to be reliable by the consensus of the public and the media in general.

Media Bias/Fact Check

This site publishes their methodology, includes their research on each page (example), and uses other well-known fact checking sites to do their research (some of them listed above). It's also quite often listed in fact checking articles to be used as a source, and listed as an equal to other fact checking sites that have been around for much longer (again, some of them listed above). This site doesn't fact check every article of each site it rates, but again, that's not reasonable due to the relatively short life of MBFC and the fact that it's all done by humans.

https://www.helenbrowngroup.com/fact-checking-the-fact-checkers/

https://www.badcredit.org/news/media-bias-fact-check-serves-as-a-watchdog-for-consumers/

https://rationalwiki.org/wiki/Media_Bias/Fact_Check

https://www.quora.com/What-is-MediaBiasFactCheck-com

https://www.kenoshanews.com/news/business/fact-checking-is-the-remedy-for-disinformation/article_7fd61aab-e459-54f9-bed4-17201682a14b.html

https://www.makeuseof.com/tag/check-political-bias-media-site/

MIT used MBFC as a baseline when creating an AI to look for media bias.

The University of Michigan Center for Social Media Responsibility also uses MBFC as a baseline for their "Iffy Quotient", which tracks factual information in Twitter and Facebook.

Against

There's plenty of criticism about MBFC, but mostly from people who think their ratings are wrong. I haven't included that, since they are obviously biased in their own favor. There's even an article written by Just Facts Daily that complains about their rating, which I included in the original Question, so I won't rehash it here.

An article by Tamar Wilner makes a snide remark about the site, calling the founder an "armchair academic" and an "armchair media analyst". On the MBFC, founder Dave Van Zandt calls himself an armchair media analyst with over 20 years experience, so I'm not sure that's such a negative. Also, Wilner's article was written over 2 years ago, so there's plenty Van Zandt and his team could have learned.

https://www.cjr.org/innovations/measure-media-bias-partisan.php

Wiltner writes about media bias and other related information, so should be considered a reliable resource.

Tamar Wilner is a Dallas-based freelance journalist and researcher who writes about misinformation, fact-checking, science communication, and all things media. https://www.cjr.org/author/tamar-wilner

The article asks if people care or whether we should even bother with rating sites, since there is no "baseline". She mentions that AI and other scientific methods to discover bias in articles are struggling and mentions a few sites that are already being used for bias checking.

The problems she mentions about MBFC (and Ad Fontes) in her article seemingly have been since addressed and hints at a bias against non-professional fact checkers, like herself. So basically she's just complaining that they are new sites that don't have a large enough human pool to pull from and their sample sizes are too small. While those are relevant criticisms, they are going to happen with any new resource, which both MBFC and Ad Fontes were at the time of the article.

As I mentioned in my Question, Wilner also writes for the Poynter Institute, which also has a negative reception of MBFC and the article was written later the same year. While I can't confirm anything, it seems likely Wiltner could have a personal influence with the writers of that 2nd article.

Conclusion

While MBFC has it's problems and critics, so do all other fact checking sites. Some of the problems come from "too small" of a sample of a site. That's understandable, since no fact checking site checks 100% of the information on the checked-site. It just means that more time and effort is needed to get a larger sample size, and with continuing checks of the sites, the ratings will become more accurate over time.

And the human factor can be significant, but that can be mitigated with double-checks by multiple people. It can also be mitigated by the rubric used to determine bias and fact checks.

With MBFC generally closely mirroring other fact checking sites that are also found to be reliable (in my Baseline Research above) and the fact that the information is updated moderately regularly, I'd give Media Bias\Fact Check the thumbs up for being a reliable source.

CYA

As much as I've presented here, I'm not infallible and may have missed something relevant. Instead of arguing in the comments, please write your own answer. If there's something wrong with my research, those comments are welcome, but only so I can fix this Answer, rather than add new sections. And because this Q&A is about media bias and reliability, consider your sources before making a claim, as it will be checked.

I've tried to use a wide range of sources without using MBFC to verify their reliability, to avoid circular logic. While I've been able to verify that most of them are reliable through third party sources (not listed in this Answer), if something isn't actually reliable, please let me know so I can address it.