Whenever possible, AllSides uses multiple methods to rate media bias, including blind bias surveys, third party data, independent reviews, and editorial reviews. The editorial review is one of our most robust methodologies. The purpose of the AllSides editorial review is to determine media bias as it reflects the subjective judgements of people across the political spectrum. An Editorial Review is when the AllSides team — which includes people from across the political spectrum — reviews the works of a source and comes to a general consensus on its bias.
The overarching question that an AllSides editorial review of a media outlet seeks to answer is: Taking into account the numerous ways that a media outlet can display preferences, biases, and perspectives, what is the overall political bias of this outlet? Overall, would someone reading this outlet generally receive a Left, Lean Left, Center, Lean Right, or Right perspective?
Editorial Review Process
Our editorial reviews are designed to be multi-partisan. Each person who is a part of an AllSides editorial review has self-rated their bias using our bias rating quiz. A group of six people or more from across the political spectrum gathers and reviews the works of a media outlet. (For Small Group Editorial Reviews, we use a group of three people on the left, center and right). We look at the media outlet’s homepage, recent articles, photos, and homepage screenshots and content dating as far back as six months using online archival tools like the Wayback Machine. In the weeks or days leading up to an editorial review meeting, the team periodically reviews the works of the outlet on their own time.
There is no discussion at this stage; each member of the editorial review team looks at the works of the media outlet on their own, and privately determines what they believe the bias of the outlet to be.
After everyone has made their individual decisions, the group convenes and shares what they believe the outlet’s bias to be, including how they arrived at that decision. Taking into account all perspectives, the team collectively decides what the media outlet’s rating should be.
Editorial Review Considerations
While AllSides does not use a formalized grading rubric to assess media bias, our participants are trained to understand and consider the 11 types of bias outlined in our guide. Some of the types of bias we take into consideration during editorial reviews are as follows:
- Spin — Is the outlet using vague language to “spin” the story in a particular direction? Does the language favor one political ideology, perspective or viewpoint over another?
- Sensationalism and emotionalism — Are words or phrases being used to provoke a strong emotional reaction in the reader or to create an illusion of drama?
- Bias by placement — What stories are being most prominently displayed on the homepage or in the articles? What facts, opinions or perspectives are most prominent?
- Photo choice — How do the photos make the subject appear? Angry? Defiant? Intelligent? Brutish? How do the photos make a situation appear? Violent? Peaceful? Chaotic? Are photos chosen to give a favorable or unfavorable view of political operatives from one side?
- Story choice — Is the outlet focusing on stories, angles, and perspectives that are generally more important or of interest to those on the left or right? What are they highlighting, and what impression does it give?
- Bias by omission — What isn’t being reported on? What details, facts, or perspectives are being omitted?
- Word choice — Are opinion statements being presented as fact? How is an issue being framed to favor a particular partisan viewpoint, perspective or opinion?
- Slant — In what light is a media outlet displaying an event, opinion, or person? Are facts, quotes, sentiments, or opinions cherry-picked to favor one side? Who is being interviewed for the story, and what perspectives are provided or omitted?
On our source pages, AllSides provides readers with information about media outlets, such as who owns the company, their political leanings, the political leanings of its audience, and the outlet’s history of bias-related controversies. We offer this information in the interest of transparency, but our editorial reviews only evaluate the content of the outlet, not its history or the partisan leanings of its ownership, staff or audience. Our editorial reviews seek to answer the question of what bias or perspective someone might receive simply from reading the outlet, not necessarily knowing its history, ownership, audience, or funding.
The Limits of the AllSides Editorial Review
As with all consensus-making processes, there may be someone on the editorial review team who disagrees with the majority decision. The majority of the team may agree that a media outlet is Lean Left, while one team member firmly believes it deserves a Left rating. Or perhaps the team has a difficult time deciding if the outlet is Lean Right or Right, as the content appears on the border. These are subjective judgments, and no one is necessarily right or wrong. AllSides is transparent and provides details about these nuances and disagreements on the source page for the outlet.
There is no “accurate” measure of bias. Bias is in the eye of the beholder and ultimately subjective. This is why, whenever possible, AllSides uses multiple methodologies to rate bias.
In addition, AllSides rates online content only, not broadcast, TV, video, or radio content unless otherwise noted. People who view AllSides Media Bias Ratings hoping to understand the bias of NPR radio news, for example, will find AllSides provides ratings for NPR’s online written content only. This is an important distinction as the bias of an outlet’s radio, TV, or broadcast content may differ significantly from its online content.
In addition, some perspectives and beliefs may be missing from the group of people who make up the editorial review team. While we may have a Democrat on the editorial review team, perhaps they are a moderate rather than a progressive Democrat. Perhaps we have a centrist or right-winger on the team, but not an explicit libertarian. Our editorial review team cannot include people from every single political ideology, as there are so many different ideologies; political thought is complex and, by nature, difficult to quantify or place into neat boxes. This is part of why AllSides solicits community feedback on our bias ratings from our audience, which includes a nearly even split of people who consider themselves left, center, and right, and encompasses various ideologies. Community feedback does not determine our ratings, but acts as a signal that a bias rating may be off and trigger more research and review.
Rating Opinion Content
If the overall bias of a source’s editorial/opinion page differs significantly from the bias of its news section, we often provide a separate bias rating for the opinion page. For example, AllSides has provided separate bias ratings for NPR online news opinion, Fox News opinion, and the New York Times opinion page. We also rate a number of individual commentators/opinion writers individually.
The opinion/editorial page media bias rating takes into account both the overall bias of the source’s editorial board and the paper’s individual opinion page writers. The editorial board’s bias is weighted, and affects the bias rating by roughly 60%. For example, The New York Times has a range of individual Opinion page writers, who have a range of biases. The New York Times Editorial Board has a Left bias as of the time of this writing. Taking all of this into account, overall, we give the New York Times Opinion Page a Left media bias. This rating includes both the overall biases of the individual writers and the Editorial Board.
Why Editorial Reviews?
Some grading systems and rubrics can be unreliable. While rubrics can be helpful, they generally assign weights to different attributes, and predefined weights introduce bias of their own and undervalue less tangible attributes.
For example, suppose a media outlet published a story and used no photo, or a very standard stock photo that didn’t show bias, and coupled it with an extremely sensationalist or biased headline. In a numerical rating system in which bias is rated from 0 to 10 with 10 being the most biased, the photo may be rated “0” on the bias scale, and the headline perhaps an “8.” The average of these ratings is 4, indicating the piece has moderate bias. However, most people would agree the sensationalist headline significantly overrules a neutral photo.
Due to the limitations of rubrics, our approach assesses the overall impact of all elements of a story, taken together, and reflects the bias and imprecise nature of those things. With so many elements to consider — word choice, story choice, sensationalism, bias by omission — we find it is better to use the subjective judgement of Americans rather than a rigid system that may turn up ratings that do not reflect how a human being might perceive the article or media outlet.
Bias is contextual. Understanding what other media outlets are covering at the same time, and how, allows us to be cognizant of the full landscape. We can see what any given media outlet is covering, omitting, highlighting, or how they are framing an issue. We review news from other sources to get a sense of the entire media landscape, so that we can more easily identify bias in a broad context.Feedback or questions? Contact AllSides at firstname.lastname@example.org.