AllSides Media Bias Ratings™ are based on scientific, multipartisan analysis. We use our patented media bias detection and display technology to drive what is arguably the world's most effective and up-to-date media bias detection engine. Our ratings reflect the average view of people across the political spectrum, not one individual.
We use multiple methods to rate bias. On the media bias rating page for any given source, you'll see the methods we used for that rating listed with a check mark next to them. Learn more about each of our methods below.
✔ Editorial Review
An Editorial Review is when the AllSides team — which includes people from the left, center and right — reviews the works of a media outlet and comes to a general consensus on its bias.
Editorial Reviews are designed to be multipartisan. Since bias is largely in the eye of the beholder, each person will have a subjective opinion of the bias of any given outlet.
We look for the common types of media bias such as slant, spin, sensationalism, and story choice. We review the outlet’s homepage, headlines, recent articles, photos, and other content dating as far back as six months using online archival tools like the Wayback Machine.
✔ Blind Bias Survey
A Blind Bias Survey is when people from all sides of the political spectrum read headlines and articles from a media outlet and provide an overall bias rating for the source, without knowing what source they are rating.
Blind Bias Surveys factor in the following:
- The political leaning of each survey participant.
- How each participant rates the bias of headlines & content from the media outlet.
- How participants with different biases (aka "Bias Groups") rate the content differently.
- The average rating from all participants across the political spectrum.
We remove all branding and identifying information from the content so that respondents are not influenced by preconceived notions of an outlet’s bias. We then look at how every "bias group" (people who self-identified as Left, Lean Left, Center, Lean Right or Right using our Rate Your Bias Tool) rated the source’s content on average. We then calculate an average bias rating across all groups.
Blind Bias Surveys are one of our most robust methods, but they do have limitations. Respondents assess a relatively small snapshot of the source's content in time, and the surveys don't include photos, content published over a long period of time, or other visual elements. This is why whenever possible, we use multiple methods to rate bias.
Below are the results from our Nov. 2020 Blind Bias Survey. View the full report here.
✔ Independent Research
Independent research or an independent review is when an AllSides editor (or sometimes, multiple editors) reviews content from a source and comes to a general conclusion on its bias. This is our lowest level of analysis.
The editor looks for common types of media bias, determines whether the outlet is transparent about its bias, and reviews any reports about the source’s political leanings. This method is typically used to provide an initial bias rating before more robust methods can be applied, or when the bias of an outlet is relatively easy to discern because the outlet is transparent about its political leaning.
✔ Third-Party Data
We sometimes take into consideration academic research, surveys, or analysis from third parties that are transparent about their system for determining bias. AllSides always notes which analyses it has taken to account on the source page for any outlet.
✔ Community Feedback
You can vote on whether or not you agree with our ratings, but our ratings are not determined by community votes or majority rule. Community feedback is valuable because it acts as a warning sign that our current bias rating may be off, triggering deeper review.
This feedback is not blind. When people rate sources, they can see which source they are rating, how AllSides has previously rated the source, and how other people have voted. Since voting is not blind, it is subject to bias itself, as users are influenced by their preconceived notions or other people’s opinions.
Community feedback is not normalized. In other words, the raw data has not been balanced to fairly represent the various segments and groups in America. For example, if 100% of the people who rated a source were Liberal, the media bias ratings they give would be very biased toward their own point of view, and not representative of America overall. A news outlet that Americans might on average consider “Left” might instead be rated by this Left group as “Center,” because the source more closely reflects their own views.
If a lot of people disagree with one of our bias ratings, it does not prompt us to outright change the bias rating, but rather, may trigger us to conduct more robust blind surveys, editorial reviews, or other kinds of research to sharpen the bias rating.
AllSides shows raw votes by our users, with the first number representing the people who agree with our bias rating, and the second number represent those who disagree. We color code these for easy reference:
It is not unusual for more than half of people to say they disagree with an AllSides Media Bias Rating, but the average rating they provide for the source is often in line with the AllSides Media Bias Rating. For example, AllSides may have rated a source as “Lean Left,” and the majority of voters say they disagree with that rating. But more than half will say they believe that source is “Left,” and the rest will rate it as “Center” or “Lean Right." That means the average will equal “Lean Left," thus supporting the original AllSides Media Bias Rating. See notes for each source for more details.
Sometimes, you will see a media bias rating to the right of this community feedback indicator. This is the average rating of people who disagreed with the AllSides media bias ratings and who bothered to also submit what they think the rating is (many disagree but do not rate themselves). It is not the average rating of the community at large since it does not include those who agree with the AllSides rating, those who disagree but did not rate, and is not normalized to reflect all of America (so one group can easily be over-represented, skewing the results.)
We rate our confidence in any given rating as Low or Initial, Medium, or High. Our confidence level is based on two things:
- How many methods have been applied, and which ones. The more methods that have been applied, the more confident we are in a rating. For example, we have more confidence in a rating for a source that has undergone both a Blind Bias Survey and an Editorial Review than one that has only undergone an Independent Review.
- The strength and consistency of the data. The stronger and more consistent the data is, the higher our Confidence Level. For example, if an outlet self-describes as “conservative,” that is a strong indication our rating is on target. Or, if an Editorial Review indicated a source was Right, but a Blind Survey returned a Lean Right rating, the inconsistency means our Confidence Level is lower. Or, if we are taking academic Third Party Data into consideration, but the research is 20 years old, our confidence level is lower since the data is so old.
How You Can Participate
We are constantly evolving the media bias rating engine. The more you participate, the better our media bias ratings will be, and the more sources we can rate.
- Discover your own bias. Make the world a better place by understanding and sharing your own bias openly! Use our bias rating tool to clarify how you judge your own bias.
- Participate in our next Blind Bias Survey. By participating in our online research, you'll be helping AllSides to make media bias transparent and help people to identify diverse perspectives — creating a stronger society as a result. Sign up to participate.
- Vote on the Media Bias Ratings. Search through our full list of ratings and let us know where you agree or disagree.
- Consume a balanced news diet. Our ratings drive our newsfeed, where we display the day’s top news stories from the Left, Center and Right of the political spectrum — side-by-side so you can see the full picture. Read our Balanced Newsfeed.
- Use the ratings in your research or product. AllSides Media Bias Ratings™ are licensed under a Creative Commons BY-NC 4.0 License, which means may be used for research or noncommercial purposes with attribution. Commercial use is permissable with a license agreement. To request this data as a CSV or JSON file, contact us.