During an AllSides Editorial Review, a multipartisan panel of six or more people across the political spectrum, with equal representation of people from the left, center, and right, review news reporting from a news source, look for indicators of media bias, and come to a general consensus on the source's bias. Our panelists are trained to spot common types of media bias.
Whenever possible, AllSides uses multiple methods to rate media bias, including Editorial Reviews, Blind Bias Surveys, third party data, independent reviews. The Editorial Review is one of our most robust methodologies for determining AllSides Media Bias Ratings™.
The purpose of the AllSides Editorial Review is to determine media bias as it reflects the subjective judgements of people across the political spectrum. The overarching question that an AllSides Editorial Review of a media outlet seeks to answer is: Taking into account the numerous ways that a media outlet can display preferences, biases, and perspectives, what is the overall political bias of this outlet? Overall, would someone reading this outlet generally receive a Left, Lean Left, Center, Lean Right, or Right perspective?
AllSides assesses content only — we do not determine media bias ratings by taking into account who owns or funds the media outlet or what agenda we believe its owners or editors may have. This can be important information and is included on our source pages, but our ratings reflect the bias of content only.
Editorial Review Process
AllSides Editorial Reviews are conducted by a multipartisan panel. Each panelist has self-rated their bias using our bias rating quiz. A group of six to nine panelists, with equal representation of people from the left, center, and right, gathers and reviews the works of a media outlet, looking for indicators of media bias. (During a Small Group Editorial Review, we use a smaller panel of three people representing the left, center and right, but the process described below is the same.)
The panel reviews multiple elements, with special attention given to the media outlet’s homepage, politics section, most recent articles, photos, story placement and story choice, word choice, and homepage screenshots and content dating as far back as six months using online archival tools like the Wayback Machine. In the weeks or days leading up to the formal review, members of the panel may periodically review the works of the outlet on their own time.
The Editorial Review consists of three stages:
- Individual Review — Each member of the panel reviews news reports from the outlet independently, taking note of media bias indicators, and privately determines what they believe the bias of the outlet to be. There is no group discussion at this stage.
- Discussion and Persuasion — The team convenes and shares what they saw, what they believe the outlet’s media bias is, and how they arrived at their decision. At this stage, some panelists may be persuaded to change their minds, as members of the panel describe the types of media bias they saw or did not see from the outlet.
- Final Determination — Taking into account all perspectives, panelists individually assign a number, between -6.0 and +6.0, that they believe best represents the bias of the media outlet. The number 0 represents the exact Center, -6.0 represents bias farthest to the left, and +6.0 represents bias farthest to the right. These numerical ratings are then transcribed into a weighted average. The weighted average is determined by taking the individual ratings by reviewers, grouping them into three bias categories (Lean Left and Left, Center, Lean Right and Right), averaging results by bias, and then taking the average of the three bias groups.
The panelists then decide whether the final numerical rating is a good representation of the outlet’s overall bias. If the panelists come to a consensus on another rating instead of the one calculated by the weighted average, the weighted average rating can be overridden. In order for an override to occur, all panelists present must agree on the new numerical rating assigned. At this stage, in addition to considering the outlet's content, the panel may also take into account the results of AllSides Blind Bias Survey data on that outlet (especially if the panel's rating conflicts with the blind survey data results), third party claims of bias against the outlet, community feedback, and third party data from academics and other groups.
The bias categories for Editorial Reviews are defined as follows:
Left: -6.00 to -3.00
Lean Left: -2.99 to -1.00
Center: -0.99 to +0.99
Lean Right: +1.00 to +2.99
Right: +3.00 to +6.00
AllSides Editorial Review panelists review the graphic shown above when making their subjective determination as to where the outlet’s bias falls on this numerical scale.
Final Determination of AllSides Media Bias Ratings™
Editorial Reviews are just one methodology AllSides applies to media outlets. To determine a final AllSides Media Bias Rating, AllSides considers the results of all methodologies that have been applied to that outlet — Editorial Reviews, Blind Bias Survey data, community feedback, and third party data from academics and other groups.
During the Final Determination stage of rating a media outlet, we also determine our confidence level in the rating (initial, low, medium, or high) by considering how much data we have on the outlet and how consistent it is. Final Determinations are ultimately subjective, and are rarely needed to be done formally, as the results of our various methodologies usually match.
Editorial Review Considerations
The AllSides Editorial Review panel is trained to understand and consider common types of media bias outlined in our resource guide. Some of the types of bias we take into consideration during Editorial Reviews are as follows:
- Spin — Is the outlet using vague language to “spin” the story in a particular direction? Does the language favor one political ideology, perspective or viewpoint over another?
- Slant — Is an issue being framed to favor a particular partisan viewpoint, perspective, policy, or opinion? In what light is a media outlet displaying an event, opinion, or person? Are facts, quotes, sentiments, or opinions cherry-picked to favor one side? Who is being interviewed for the story, and what perspectives are provided or omitted?
- Sensationalism and Emotionalism — Are words or phrases being used to provoke a strong emotional reaction in the reader or to create an illusion of drama?
- Bias by Story and Viewpoint Placement — What stories are being most prominently displayed on the homepage or "above the fold", indicating the editor finds those stories most important? What facts, opinions or perspectives are most prominent or buried? Are people on both sides quoted, or only one side? Are perspectives from one side given less space or buried at the end of the article?
- Photo Choice — How do the photos make the subject appear? Angry? Defiant? Intelligent? Brutish? How do the photos make a situation appear? Violent? Peaceful? Chaotic? Are photos chosen to give a favorable or unfavorable view of political operatives from one side?
- Story Choice — Is the outlet focusing on stories, angles, and perspectives that are generally more important or of interest to those on the left or right? What are they highlighting, and what impression does it give?
- Bias by Omission — What isn’t being reported on? What details, stories, facts, perspectives, or viewpoints are being omitted?
- Word Choice — Is the outlet choosing words the left or right favors, phrases that provoke the other side, or terminology that the one side disagrees with?
- Subjective Adjectives — Is the reporter using adjectives to characterize nouns, instead of letting the reader decide what they think of events, people or situations for themselves?
- Omission of Source Attribution — Does the outlet cite sources or omit them?
- Opinion or Analysis Statements Presented as Fact — Outside of the opinion page, do the writers frequently include subjective statements based on personal opinions, assumptions, beliefs, tastes, preferences, or interpretations in their news writing?
- Labeling of News and Opinion — Does the outlet clearly label the difference in news and opinion content?
On our source pages, AllSides provides readers with additional information about media outlets, such as who owns the company, the owner's political leanings, and the political leanings of its audience. We offer this information in the interest of transparency, but our Editorial Reviews only evaluate the content of the outlet, not its history or the partisan leanings of its ownership, staff or audience. We do not give a rating based on the fact that more liberals or conservatives read that outlet, or that it is owned by a liberal or a conservative, or that it is funded by someone thought to be liberal or conservative; we assess content only. Editorial Reviews seek to answer the question of what bias or perspective someone might receive simply from reading the outlet without necessarily knowing its history, ownership, audience, or funding.
AllSides does not use a formalized grading rubric to assess media bias, as we believe rubrics can introduce unintended bias themselves as well as fail to capture subjective elements adequately.
The Limits of the AllSides Editorial Review
As with all consensus-making processes, there may be someone on the panel who disagrees with the majority decision. The majority of the team may agree that a media outlet is Lean Left, while one team member firmly believes it deserves a Left rating. Or perhaps the team has a difficult time deciding if the outlet is Lean Left or Center, as the content appears on the border. These are subjective judgments, and no one is necessarily right or wrong. AllSides is transparent and provides details about these nuances and disagreements on the source page for each outlet.
There is no “accurate” measure of media bias. Bias is in the eye of the beholder and ultimately subjective. This is why, whenever possible, AllSides uses multiple methodologies to rate bias.
Unless otherwise noted, AllSides Media Bias Ratings™ are based on online content only, not broadcast, TV, video, or radio content. People who view AllSides Media Bias Ratings™ hoping to understand the bias of NPR radio news, for example, will find AllSides provides ratings for NPR’s online, written content only. This is an important distinction, as the bias of an outlet’s radio, TV, or broadcast content may differ significantly from its online content.
In addition, some perspectives and beliefs may be missing from the group of people who make up the panel. While we may have a Democrat on the Editorial Review team, perhaps they are a moderate rather than a progressive Democrat. Perhaps we have a centrist or a traditionalist conservative on the team, but not an explicit libertarian. Our Editorial Review team cannot feasibly include people from every single political category, as there are so many different ideologies; political thought is complex and has endless variation. This is part of why AllSides solicits community feedback on our bias ratings from our audience, which includes a nearly even split of people who consider themselves left, center, and right, and encompasses various ideologies. Community feedback does not determine our ratings, but acts as a signal that a bias rating may be off and trigger more research and review.
Rating Opinion Content
If the overall bias of a source’s editorial/opinion content differs significantly from the bias of its news section, we often provide a separate media bias rating for the opinion page. We also rate a number of individual commentators/opinion writers individually.
The opinion/editorial page media bias rating takes into account both the overall bias of the source’s editorial board and the paper’s individual opinion page writers. The editorial board’s bias is weighted, and affects the bias rating by roughly 60%. For example, The New York Times has a range of individual opinion writers, who have a range of biases. Meanwhile, The New York Times Editorial Board has a Left bias as of the time of this writing. Taking all of this into account, overall, we give the New York Times Opinion Page a Left media bias, even though some conservative writers may be present. This rating includes both the overall biases of the individual writers and the Editorial Board, but weights the Editorial Board.
About Video Editorial Reviews
AllSides Video Editorial Reviews are designed to be multi-partisan in the same way as Written Editorial Reviews, with each person self-rating their bias using our bias rating quiz. A group of six people or more from across the political spectrum gathers and watches a media outlet’s broadcast content. For Small Group Editorial Reviews, we use a group of three people representing the left, center and right. We then follow the same process as an Editorial Review of written content.
Like with our analysis of written content reviews, our participants are trained to understand and consider the types of media bias outlined in our guide. We look for things such as sensationalism, editorialization, the inclusion of subjective adjectives and qualifiers, story choice, and other elements.
For Video Editorial Reviews, we only evaluate the content of the outlet, not its history or the partisan leanings of its ownership or audience. AllSides Video Editorial Reviews seek to answer the question of what bias or perspective someone might receive simply from watching the outlet, not necessarily knowing its history, ownership, audience, or funding.
Why Editorial Reviews?
Some grading systems and rubrics can be unreliable. While rubrics can be helpful, they generally assign weights to different attributes, and predefined weights introduce bias of their own and undervalue less tangible attributes.
For example, suppose a media outlet published a story and used no photo, or a very standard stock photo that didn’t show bias, and coupled it with an extremely sensationalist or biased headline. In a numerical rating system in which bias is rated from 0 to 10 with 10 being the most biased, the photo may be rated “0” on the bias scale, and the headline perhaps an “8.” The average of these ratings is 4, indicating the piece has moderate bias. However, most people would agree the sensationalist headline significantly overrules a neutral photo.
Due to the limitations of rubrics, our approach assesses the overall impact of all elements of a story, taken together, and reflects the bias and imprecise nature of those things. With so many elements to consider — word choice, story choice, sensationalism, bias by omission, etc. — we find it is better to use the subjective judgement of Americans rather than a rigid system that may turn up ratings that do not reflect how a human being might perceive the article or media outlet.
Bias is subjective and contextual. Understanding what other media outlets are covering at the same time, and how, allows us to be cognizant of the full landscape. We can see what any given media outlet is covering, omitting, highlighting, or how they are framing an issue. We review news from other sources to get a sense of the entire media landscape, so that we can more easily identify bias in a broad context.
Feedback or questions? Contact AllSides at feedback@allsides.com.