Friday, August 3, 2018

Voters in 2016 asked for a Democratic Senate by the highest margin in decades. They didn't get one.

The United States Senate is an unusual place. Unlike the House of Representatives, known as the "People's House," the Senate was designed to reflect the sovereign interests of the states themselves, rather than the people living in them. (Prior to the adoption of the 17th Amendment in 1913, in most states, the people didn't even get to weigh in on the Senators representing them; they were selected by state legislatures.) By assigning two Senate seats to each state, regardless of population, the framers intended to balance the interests of smaller states with those of larger ones.

These concerns remain valid today, as less populated states now tend to be largely rural, and more populated states more urbanized, affecting their legislative interests in important ways. Nonetheless, modern circumstances have skewed the impact of this initial compromise. For one thing, the disparity in population between the smallest and largest states has grown drastically, giving small states immense power in the Senate relative to their population. Additionally, the move to popular election of Senators - and the expansion of voting rights to all adult citizens, rather than land-owning white males - has led to an expectation that the Congress ought to be more responsive to the people's wishes.

Measuring the people's preferences on a national level, however, has always been tricky when it comes to the Senate. Unlike House or Presidential elections, the nation doesn't all vote at once, giving us tidy popular vote totals that we can compare to the actual results (in House seats or Presidential electoral votes). Senate seats are divided into three classes, serving staggered six-year terms, and only two-thirds of the states have a Senate race in any given election year, so the popular vote from one election cycle cannot be held to represent the wishes of the nation as a whole.

We can gain a better understanding, however, by combining the votes from the most recent election of each Senate seat, over the last three election cycles. This ensures that each voter is counted equally in the totals, regardless of the state they live in. It has the added advantage of including two data points from each state (one for each Senate seat), usually from two points in time. This will smooth out some of the outlier effects from wave elections, as well as the effects of candidates who are far more popular in their state than their party. (For example, a West Virginian who votes for Democrat Joe Manchin isn't necessarily doing so because they want Chuck Schumer to be Majority Leader, though they may be contributing to that. But the same state's votes for Republican Shelley Moore Capito will cancel out some of that dissonance.)

To determine the Aggregate Popular Vote - let's call it APV for short - for the current Senate, we would total up the votes from the November 2012 election (for all Class I seats), November 2014 (for all Class II seats except for Alabama's), and November 2016 (for all Class III seats), as well as the December 2017 special election for Alabama's Class II seat. This reveals a huge disparity between the votes and the result:

Democratic
Votes
Republican
Votes
Independent/3rd
Party/Other
MarginDem SeatsRep SeatsFloor Margin
121,741,009102,034,2439,152,897D+8.46%4751R+4

Even when you include Angus King and Bernie Sanders, the two Independent Senators who caucus with the Democrats, Republicans control the chamber with a two-seat advantage, despite the voters' preference for Democrats by a wide margin. That margin should be viewed with a very large asterisk, though, which I will explain further on.

This lopsided victory for Republicans is not an unusual situation, given the outsized power and predominantly conservative politics of the smaller, more rural states. I have collected historical election results that allow us to determine the APV of the Senate at any point since 1987. (In order to capture the effects of special elections, appointments, and Senators who changed party affiliation during their terms - but only weight these events by the amount of time they were in effect - I've followed the trends of APV on a month-by-month basis, rather than just every two years.) Over the 31 years from 1987 to 2018, Democrats have received a plurailty of the APV a whopping 94% of the time, but have only controlled the chamber (even including Independents caucusing with them) 56% of the time. The APV has only favored Republicans for two years, 1999-2000, out of the entire 31-year period.

The Republican structural advantage is such that, over the last three decades, Republicans have controlled the Senate with a losing APV 38% of the time, but Democrats have never won control after losing the APV.

But the lopsided result of the 2016 electionwould seem to far exceed even these historical trends. The average APV since 1987 has been in favor of the Democratic Party by a 2.94% margin. The APV margin after the 2016 election was D+8.13%, a record high during the period we have data for. It increased slightly, to D+8.46%, with the 2017 special election victory of Doug Jones in Alabama.

How did the margin increase so heavily toward Democrats in a year that overall went so poorly for them? A major part of it was the races for the large, liberal states of California and New York, which had higher than usual turnout. California, in particular, was able to run up the tallies because of their peculiar "jungle primary" election system. During the primary election, all candidates are listed on the same ballot, regardless of party, and the top two vote-getters go on to the general election. In 2016, the top two candidates in the primary were both Democrats, and faced each other in November - meaning that no Republican appeared on the ballot, and every vote cast was for a Democrat. Other Senate elections in past years have occurred where one major party did not field a candidate - more often with a Republican running unopposed than a Democrat - but never in a state as populous as California.

To isolate that effect, we need to estimate what the APV would be if a Republican had appeared on the California ballot. In the 2016 California primary, twelve Republican candidates received a combined 28.6% of the vote, while seven Democrats received 64.1% (with 7.3% going to other candidates and write-ins). If we assume that the general election would have mirrored those results, had the ballot included a Republican and independent candidate, we would see Democrats receive about 4.4 million fewer votes, with Republicans taking 3.5 million of them. This would drop the nationwide APV margin to D+4.73% (rising to D+5.07% after the 2017 special election).

Those theoretical margins are much closer to the usual trendline, though they are still well above average. It is also a significant swing towards Democrats compared to the pre-2016 APV (which favored Democrats by only 2.09%), and the highest margin ever in favor of Democrats in a time when Republicans still controlled the chamber. Prior to 2015, Republicans had never won control while losing the APV by more than 1.17%.

Those interested can view the data used to calculate the APV.