Whenever possible, AllSides uses multiple methods to rate media bias, including Editorial Reviews, Blind Bias Surveys, third party data, independent reviews. The Editorial Review is one of our most robust methodologies for determining AllSides Media Bias Ratings™. The purpose of the AllSides Editorial Review is to determine media bias as it reflects the subjective judgements of people across the political spectrum. During an Editorial Review, a multipartisan panel of people from across the political spectrum review the works of a source, look for indicators of media bias, and come to a general consensus on the source's bias. Our expert panelists are media professionals who are trained to spot types of media bias.
The overarching question that an AllSides Editorial Review of a media outlet seeks to answer is: Taking into account the numerous ways that a media outlet can display preferences, biases, and perspectives, what is the overall political bias of this outlet? Overall, would someone reading this outlet generally receive a Left, Lean Left, Center, Lean Right, or Right perspective?
AllSides rates content only — we do not determine media bias ratings based on who owns the media outlet, funds it, or what agenda we believe its owners or editors may have. This can be important information and is included on our source pages, but our ratings reflect content only.
Editorial Review Process
AllSides Editorial Reviews are conducted by a multipartisan expert panel. Each panelist has self-rated their bias using our bias rating quiz. A group of six experts or more, with equal representation of people from the left, center, and right, gathers and reviews the works of a media outlet, looking for indicators of media bias. (For Small Group Editorial Reviews, we use a smaller panel of three people representing the left, center and right).
The expert panel reviews multiple elements, with special attention given to the media outlet’s homepage, politics section, most recent articles, photos, story placement and story choice, and homepage screenshots and content dating as far back as six months using online archival tools like the Wayback Machine. In the weeks or days leading up to the formal review, members of the panel may periodically review the works of the outlet on their own time.
When the expert panel meets, the Editorial Review consists of three stages:
- Individual Review — Each member of the panel reviews the works of the media outlet on their own, taking note of media bias indicators, and privately determines what they believe the bias of the outlet to be. There is no group discussion at this stage.
- Discussion and Persuasion — The team convenes and shares what they believe the outlet’s media bias to be and how they arrived at their decision. At this stage, some experts may be persuaded to change their minds as members of the group describe the types of media bias they saw or did not see from the outlet.
- Editorial Review Result — Taking into account all perspectives, the panel collectively decides what the Editorial Review result is. Often, there will be perfect consensus; other times, there will be some panelists who disagree with the final result.
Final Determination of Media Bias Ratings
Editorial Reviews are just one methodology AllSides applies to media outlets. To determine a final AllSides Media Bias Rating, AllSides considers the results of all methodologies that have been applied to that outlet — Editorial Reviews, Blind Bias Survey data, third party claims of bias against the outlet (if AllSides has determined they are credible claims), community feedback, and third party data from academics and other groups.
During the Final Determination stage of rating a media outlet, we also determine our confidence level in the rating (initial, low, medium, or high) by considering how much data we have on the outlet and how consistent it is. Final Determinations are ultimately subjective, and are rarely needed to be done formally, as the results of our individual methodologies usually match.
Editorial Review Considerations
The AllSides expert Editorial Review panel is trained to understand and consider common types of media bias outlined in our resource guide. Some of the types of bias we take into consideration during Editorial Reviews are as follows:
- Spin — Is the outlet using vague language to “spin” the story in a particular direction? Does the language favor one political ideology, perspective or viewpoint over another?
- Slant — Is an issue being framed to favor a particular partisan viewpoint, perspective, policy, or opinion? In what light is a media outlet displaying an event, opinion, or person? Are facts, quotes, sentiments, or opinions cherry-picked to favor one side? Who is being interviewed for the story, and what perspectives are provided or omitted?
- Sensationalism and Emotionalism — Are words or phrases being used to provoke a strong emotional reaction in the reader or to create an illusion of drama?
- Bias by Story and Viewpoint Placement — What stories are being most prominently displayed on the homepage or in the articles? What facts, opinions or perspectives are most prominent or buried? Are people on both sides quoted, or only one side? Are perspectives from one side given less space or buried at the end of the article?
- Photo Choice — How do the photos make the subject appear? Angry? Defiant? Intelligent? Brutish? How do the photos make a situation appear? Violent? Peaceful? Chaotic? Are photos chosen to give a favorable or unfavorable view of political operatives from one side?
- Story Choice — Is the outlet focusing on stories, angles, and perspectives that are generally more important or of interest to those on the left or right? What are they highlighting, and what impression does it give?
- Bias by Omission — What isn’t being reported on? What details, facts, perspectives, or viewpoints are being omitted?
- Word Choice — Is the outlet choosing words that the left or right favors?
- Subjective Adjectives — Is the reporter using adjectives to characterize nouns, instead of letting the reader decide what they think of events, people or situations for themselves?
- Omission of Source Attribution — Does the outlet cite sources or omit them?
- Opinion Statements Presented as Fact — Outside of the opinion page, do the writers frequently include subjective statements based on personal opinions, assumptions, beliefs, tastes, preferences, or interpretations in their writing?
On our source pages, AllSides provides readers with additional information about media outlets, such as who owns the company, the owner's political leanings, and the political leanings of its audience. We offer this information in the interest of transparency, but our Editorial Reviews only evaluate the content of the outlet, not its history or the partisan leanings of its ownership, staff or audience. We do not give a rating based on the fact that more liberals or conservatives read that outlet, or that it is owned by a liberal or a conservative; we assess content only. Editorial Reviews seek to answer the question of what bias or perspective someone might receive simply from reading the outlet without necessarily knowing its history, ownership, audience, or funding.
AllSides does not use a formalized grading rubric to assess media bias, as we believe rubrics can introduce unintended bias themselves as well as fail to capture subjective elements adequately.
The Limits of the AllSides Editorial Review
As with all consensus-making processes, there may be someone on the expert panel who disagrees with the majority decision. The majority of the team may agree that a media outlet is Lean Left, while one team member firmly believes it deserves a Left rating. Or perhaps the team has a difficult time deciding if the outlet is Lean Left or Center, as the content appears on the border. These are subjective judgments, and no one is necessarily right or wrong. AllSides is transparent and provides details about these nuances and disagreements on the source page for each outlet.
There is no “accurate” measure of media bias. Bias is in the eye of the beholder and ultimately subjective. This is why, whenever possible, AllSides uses multiple methodologies to rate bias.
In addition, AllSides rates online content only, not broadcast, TV, video, or radio content, unless otherwise noted. People who view AllSides Media Bias Ratings™ hoping to understand the bias of NPR radio news, for example, will find AllSides provides ratings for NPR’s online written content only. This is an important distinction, as the bias of an outlet’s radio, TV, or broadcast content may differ significantly from its online content.
In addition, some perspectives and beliefs may be missing from the group of people who make up the expert panel. While we may have a Democrat on the editorial review team, perhaps they are a moderate rather than a progressive Democrat. Perhaps we have a centrist or a traditionalist conservative on the team, but not an explicit libertarian. Our editorial review team cannot include people from every single political ideology, as there are so many different ideologies; political thought is complex and, by nature, difficult to quantify or place into neat boxes. This is part of why AllSides solicits community feedback on our bias ratings from our audience, which includes a nearly even split of people who consider themselves left, center, and right, and encompasses various ideologies. Community feedback does not determine our ratings, but acts as a signal that a bias rating may be off and trigger more research and review.
Rating Opinion Content
If the overall bias of a source’s editorial/opinion content differs significantly from the bias of its news section, we often provide a separate media bias rating for the opinion page. We also rate a number of individual commentators/opinion writers individually.
The opinion/editorial page media bias rating takes into account both the overall bias of the source’s editorial board and the paper’s individual opinion page writers. The editorial board’s bias is weighted, and affects the bias rating by roughly 60%. For example, The New York Times has a range of individual opinion writers, who have a range of biases. Meanwhile, The New York Times Editorial Board has a Left bias as of the time of this writing. Taking all of this into account, overall, we give the New York Times Opinion Page a Left media bias, even though some conservative writers may be present. This rating includes both the overall biases of the individual writers and the Editorial Board, but weights the Editorial Board.
About Video Editorial Reviews
AllSides Video Editorial Reviews are designed to be multi-partisan in the same way as regular Editorial Reviews, with each person self-rating their bias using our bias rating quiz. A group of six people or more from across the political spectrum gathers and watches a media outlet’s broadcast content. For Small Group Editorial Reviews, we use a group of three people representing the left, center and right. We watch the outlet’s broadcast news coverage separately, then come together to discuss what we saw and what we think the bias rating is.
Our participants are trained to understand and consider the types of media bias outlined in our guide. We look for things such as sensationalism, editorialization, the inclusion of subjective adjectives and qualifiers, story choice, and other elements. AllSides does not use a formalized grading rubric to assess media bias, as we believe rubrics can introduce unintended bias themselves as well as fail to capture subjective elements adequately.
In Video Editorial Reviews, we only evaluate the content of the outlet, not its history or the partisan leanings of its ownership or audience. AllSides Video Editorial Reviews seek to answer the question of what bias or perspective someone might receive simply from watching the outlet, not necessarily knowing its history, ownership, audience, or funding.
Why Editorial Reviews?
Some grading systems and rubrics can be unreliable. While rubrics can be helpful, they generally assign weights to different attributes, and predefined weights introduce bias of their own and undervalue less tangible attributes.
For example, suppose a media outlet published a story and used no photo, or a very standard stock photo that didn’t show bias, and coupled it with an extremely sensationalist or biased headline. In a numerical rating system in which bias is rated from 0 to 10 with 10 being the most biased, the photo may be rated “0” on the bias scale, and the headline perhaps an “8.” The average of these ratings is 4, indicating the piece has moderate bias. However, most people would agree the sensationalist headline significantly overrules a neutral photo.
Due to the limitations of rubrics, our approach assesses the overall impact of all elements of a story, taken together, and reflects the bias and imprecise nature of those things. With so many elements to consider — word choice, story choice, sensationalism, bias by omission, etc. — we find it is better to use the subjective judgement of Americans rather than a rigid system that may turn up ratings that do not reflect how a human being might perceive the article or media outlet.
Bias is subjective and contextual. Understanding what other media outlets are covering at the same time, and how, allows us to be cognizant of the full landscape. We can see what any given media outlet is covering, omitting, highlighting, or how they are framing an issue. We review news from other sources to get a sense of the entire media landscape, so that we can more easily identify bias in a broad context.Feedback or questions? Contact AllSides at firstname.lastname@example.org.