Progressives hoping that the nation would be cleansed of President Trump and the GOP that enabled him in a mandate...
talkingpointsmemo.com
What many people simply don't understand is that the polilng industry only makes statements about the probability of a certain outcome to happen, nothing more and nothing less. It's not that they are saying "this will happen", but "within a 95% level of confidence this will likely happen." - so there's still quite a margin of error. So this means when predicting like e.g. 50.3% for one candidate it means in reality e.g. 50.3% means a range from 49.04% up to 51.55% (+/- 2.5%). Good polling institutes will always disclose their ranges, but most of the times you will never see them on television, but only this one value because statistically it is the most likely one to happen. The rest is the margin of error, which still might happen though.
And getting to higher levels of confidence in theory is possible, in reality too hard to achieve and expensive. This is why public polls settled for 95% a long time ago.
The other thing is a methodical problem, namely that you really need a good enough random sample at hand to ask your questions to. If you fail to construct a good enough random sample, e.g. by leaving out certain parts of society, then you've got a heavy statistical and your results will become biased as well. Meaning: they are still correct for the random sample you drew them from, but since the society looks different for the whole society probably meaningless and simply wrong.
The textbook example for such a wrong approach is the public poll of the 1936 election: straw polls back then were the method of choice, but the people doing the straw polls were middle class and above only, owning cars and telephones. Newcomer Gallup instead used a much smaller based quota sample, and was the only one whose prediction on the outcome was right because in his quota sample he had the working poor the rest had uncovered. The same method though that failed greatly in 2016.
The problem is that the pollsters can only work within the budget and tools they've got. Constructing a really meaningful, good random sample got harder and harder over the decades, because the country became much more diverse and torn. To poll these peoples you need to be able to get in touch with them, which is not always easy. But then again your statements can only be as good as the random sample you draw your data from.
The other thing is that really many voters are undecided today about how they are going to vote until they are before the ballot. So maybe they are telling A to the pollster institute and voting B or vice versa. In theory this could happen equally for both parties, so normally this should null itself somewhat out.
The problem though is with someone as Trump as head of state, many of supporters were probably motivated to lie to the pollsters. And that's a thing you can only hardly come by, because how should you know that they are lying? So those people could really, if being enough, put poison the data sample one way or another.
The other thing the polls simply cannot forecast are the sum of all events trying to sabotage the election, may it the slower working USPS, pulling people out of the voting registers and whatnot. And trying to manipulate the election at large scale by the parties sadly is something in the USA which is not the exception, but the rule. But this is not something the pollsters can fix, because they need to assume that all votes casted are getting counted. Sabotaging the election on small enough tipping points can yield to a very much different outcome in the maze of what is the presidential election in the USA. There's been a video around on Youtube somewhere, demonstrating it only needs really minor manipulations around +/- 50000 votes in 2 or 3 rural states to change the whole outcome. Sadly I cannot find it at the moment.
What also might in theory influence the outcome of an election is the publishing of polling results before voting day, it might lead to a bandwagon or underdog effect. As far as I know there is not enough empirical evidence for this yet, but it is plausible.
So to sum it up the polling stuff is quite hard to get right, because there are many moving parts involved and it is easy to make some minor errors which might lead to quite nasty biased results. And many people simply do get it wrong what a public poll really is all about.