What do the media bias ratings mean?
These are subjective judgements made by people across the country. Here is our rough approximation for what the media bias ratings mean:
Note that "Center" does not always mean unbiased, neutral or reasonable, just as "far Left" and "far Right" do not always mean "extreme" or "unreasonable." Learn more about what a Center media bias rating means here. Think of our bias ratings as points of view, each providing pieces of the puzzle, to help you gain a more holistic view.
How AllSides Calculates Media Bias
AllSides rates media bias using a patented media bias detection and display technology that drives what is arguably the world's most effective and up-to-date media bias detection engine. It's powered by the best statistical research and methodologies. We've rated the bias of over 800 media outlets and writers.
You help to drive the media bias ratings.
What you do at AllSides affects our media bias ratings. That includes how you rate your own bias and how you rate the bias of news sites on our Media Bias Ratings page, in our blind bias surveys, and throughout our site. All of this is taken into account in our data, which includes multiple methodologies and is statistically normalized to represent the average judgement of Americans.
AllSides uses multiple methods for calculating media bias ratings.
Our blind bias survey, described in the graphic below, represents one of our most robust media bias rating methods, but it is not the only method we use. A source might openly share its bias, or it may be determined by third party research, an independent review, or an editorial review. Blind surveys have limitations, because they ask respondents to assess a relatively small snapshot of the source's content in time. The surveys don't include photos, content published over a long period of time, or other elements. Therefore, we also use other methods to arrive at a final bias rating, such as Editorial Reviews, third party data and independent reviews.Take a look at the multiple methods AllSides uses to measure and rate media bias.
Our media bias detection engine gets smarter as time goes on. The more you participate, the better our bias ratings will be.
How AllSides Displays Media Bias Ratings
Throughout AllSides.com, you will find references to the media bias ratings of media outlets, writers, and more.
We rate media bias using a patented process to identify and display the average judgement of Americans. We update AllSides Media Bias Ratings and our confidence levels as more data is gathered and assessed, or as outlets change their bias over time. That means the media bias ratings are fluid and updated as new information is assessed.
See all of our media bias ratings on the AllSides Media Bias Ratings page. On a media source’s page, you’ll see:
1. The media bias rating for that source
2. How we assigned the media bias rating
3. Which methods influenced our rating
4. How confident we are in our rating of that source’s bias.
Click on the name of any media source or writer for more information. Here's what our media bias ratings look like throughout the site:
Unless otherwise noted, a media bias rating refers only to a source’s written online news articles, not the source’s TV, radio, video, or broadcast content. If the overall bias of a source’s editorial/opinion page differs significantly from the bias of its news section, we often provide a separate bias rating for the opinion page.
The Basis of the Media Bias Rating
On the media bias rating page for any given source, you'll see the methods we used for that rating listed with a check mark next to them.
- Blind Bias Survey. If this is checked, the media bias rating is driven by data from our Blind Bias Surveys. We gather readers from all parts of the political bias spectrum, and have them read and rate articles and headlines blindly — without telling them the source of the content. We then look at how every "bias group" (people on the left, center, and right) rates the source on average, and then calculate an average across all groups. AllSides normalizes this data, making sure that the survey audience accurately reflects the social and political diversity of the nation. This prevents “gaming” (when an individual or group tries to unfairly impact the ratings) or giving too much influence to any segment of the population. This robust evaluation is at the core of the patented AllSides Media Bias Ratings™ methodology.
Blind surveys do have limitations, because they ask respondents to assess a relatively small snapshot of the source's content in time. The surveys don't include photos, content published over a long period of time, or other elements. Therefore, we also use other methods to arrive at a final bias rating, described below.
- Editorial Review. If this is checked, the AllSides editorial staff — which includes people from across the political spectrum — reviewed the works of this source and came to a general consensus on its bias. These reviews always include a diversity of individuals covering the full range of political bias from left to right. We asses media outlets for the common types of media bias. Sometimes we also do "blind" editorial reviews, when our editorial staff does not know the source of the content they are rating. Read about AllSides editorial reviews in more detail here.
The overarching question that an AllSides Editorial Review seeks to answer is: Taking into account the numerous ways that a media outlet can display preferences, biases, and perspectives, what is the overall political bias of this outlet? Overall, would someone reading this outlet generally receive a Left, Lean Left, Center, Lean Right, or Right perspective?
- Third Party Analysis. If this is checked, the media bias rating has been influenced by third-party analysis or data that AllSides has determined to be credible. This may include academic research, surveys, or analysis from third parties that have a published and transparent system for evaluating the bias of multiple sources. AllSides always notes which analyses it has taken to account on the source page for any outlet.
- Independent Review. If this is checked, the bias rating is influenced by an independent review performed by AllSides. This is our lowest level of analysis. An AllSides editor, or multiple editors, reviewed content from this source and came to a general conclusion on its bias; they also investigated what the media and other sources, both partisan and nonpartisan, reported about the political leanings of this source. This method is frequently used for initial bias ratings before more robust methods can be applied, or ratings for which the bias of an outlet is relatively easy to discern.
- Community Feedback. Users can indicate whether or not they agree with our ratings, but our ratings are not determined by community votes. Instead, community feedback simply acts as a “warning system” that our current bias ratings may be off. This feedback is not blind. When people rate sources, they can see which source they are rating, how AllSides has previously rated the source, and how other people have voted. Since voting is not blind, it is subject to bias itself, as users are influenced by their preconceived notions or other people’s opinions.
Community feedback is not normalized. In other words, the raw data has not been balanced to fairly represent the various segments and groups in America. For example, if 100% of the people who rated a source were Liberal, the media bias ratings they give would be very biased toward their own point of view, and not representative of America overall. A news outlet that Americans might on average consider “Left” might instead be rated by this Left group as “Center,” because the source more closely reflects their own views.
If a lot of people disagree with one of our bias ratings, it does not prompt us to outright change the bias rating, but rather, may trigger us to conduct more robust blind surveys, editorial reviews, or other kinds of research to sharpen the bias rating accuracy.
AllSides shows the raw votes by our users, with the first number representing the people who agree with our bias rating, and the second number represent those who disagree. We color code these for easy reference:
It is not unusual for more than half of people to say they disagree with an AllSides Media Bias Rating, but the average rating they provide for the source is often in line with the AllSides Media Bias Rating. For example, AllSides may have rated a source as “Lean Left,” and the majority of voters say they disagree with that rating. But more than half will say they believe that source is “Left,” and the rest will rate it as “Center” or “Lean Right." That means the average will equal “Lean Left," thus supporting the original AllSides Media Bias Rating. See notes for each source for more details.
Sometimes, you will see a media bias rating to the right of this community feedback indicator. This is the average rating of people who disagreed with the AllSides media bias ratings and who bothered to also submit what they think the rating is (many disagree but do not rate themselves). It is not the average rating of the community at large since it does not include those who agree with the AllSides rating, those who disagree but did not rate, and is not normalized to reflect all of America (so one group can easily be over-represented, skewing the results.)
The Confidence Level in our Media Bias Rating
The Confidence Level on an AllSides Media Bias Rating page may show as Low or Initial, Medium or High, depending on how confident we are about the bias rating of the media source. We base our Confidence Level on two things:
- How many methods have been applied and which methods have been applied. The more reliable methods applied, the higher our Confidence Level. For example, we usually give more weight to our Blind Surveys and the less weight to what we call Secondary Research, which is an AllSides staff member’s initial assessment of bias based on online research.
- The strength and consistency of the data. The more strength and consistency in the bias information, the higher our Confidence Level. For example, with Secondary Research, if a journalist is self-described as “conservative” and Wikipedia labels her that way too, that shows both strength and consistency. If the Community Feedback includes thousands of users, that has more impact than feedback from fewer votes. If we take academic Third Party Data into consideration in a bias rating, but the research is 20 years old, we wouldn’t give it as much weight as more recent data. Or, if two of our assessment methods indicate the source is Right, but one of our methods indicates it’s Left, our Confidence Level in the bias rating is lower.
We are constantly evolving the media bias rating engine. The more you participate, the better our media bias ratings will be, and the more sources we can rate.
Rate your own bias to get started. We're continuing to improve ways to help you get the most accurate bias self-rating so you can participate on AllSides and in life with transparency and self-awareness. Make the world a better place by understanding and sharing your own bias openly!