This is a corrected version of an article published in Projecting Indonesia.
Being applied on a large scale for the first time in Indonesia by LP3ES in 1997, Quick Count method have been an integral part of Indonesian election. Today, there are a considerable number of survey institutions doing and publishing Quick Count. Generally, the Quick Count result is published simultaneously with some statistical lingos. Keywords such as confidence level, margin of error and sample size as well as various kinds of sampling methods name come side by side with percentages achieved by parties or candidates. Given different jargons on different survey publications, how should we read, understand, and perhaps compare these results?
A complete treatment on this will need rewriting of some part Basic Statistics textbook into this article; therefore this article does not aim for a perfect understanding but only an intuitive idea on the interplay of these key ideas. This writing seeks to be your grain of salt next time you have to take in a Quick Count result, and analyzing this mock example may fit that purpose:
Projecting Indonesia Survey carried out a Quick Count survey for Indonesia Presidential Election 2014. Candidate A achieved 39.2% and Candidate B achieved 60.8% of all legitimate votes. Sample size is greater than 500.000 people in 1000 polling stations, with 95% level of confidence and 1% margin of error.
One simple way of reading this is as follows: On the simplest assumption on the survey and data (Simple Random Sampling etc. etc.), Projecting Indonesia Survey simply calculates the percentage of votes of all peoples in all polling stations that they sampled. The number of all these people is greater than 100.000. Projecting Indonesia Survey is 95% confident that the actual result lies in 1% interval of their result; that is the exact result is somewhere between (A vs. B: 38.2% vs. 61.8%) and (A vs. B: 40.2% vs. 59.8%). Of course they acknowledge that there is 5% chance that the actual result is outside that span (A got less than 38.2% or more than 40.2%).
What does it mean by 95% confident in the above interpretation? At least for the matter of poll survey, this can be understood by saying that: If the survey is replicated multiple times with exactly the same method and condition, then 95% of the time the sampling result will lie inside the margin of error from the actual result. This can also be interpreted as: It is unlikely (5% in this case) for two survey institutions employing the same method at the same condition to have results outside each other double margin of error. Of course 5% is still 1 out of 20, so it can still happen; nonetheless results from different survey institutions that are far off from each other may indicate that something is dubious.
Confidence level is not a measure of accuracy. If any of these concepts were close to accuracy, it would be the margin of error. However, nothing can be concluded from the above information about the true accuracy and bias, and while it is tempting to comment about accuracy of a survey, it is something that should be assessed in comparison with the other Quick Counts and the actual result unless the result discussed is far too absurd (doesn’t make sense or differ very far with opinion polls etc.).
In a world where a coin has 3 sides, sample size, level of confidence, and margin of error are sides of the same coin. Intuitively, the larger our sample is, the more confident we are and vice versa. Similarly, when the sample size becomes greater, the error becomes smaller and so is the opposite. It is also straightforward to argue the if our confidence interval is wide (margin of error is high) the more confident we are and the other way around. For instance, it is easy to feel more confident to assure candidate A true percentage is somewhere in 20-80% stretch than to assure that it is inside the 40-60% interval. That is why there is a formula relating sample size, level of confidence and margin of error. Before the survey is conducted, the survey institution would usually start with level of confidence and margin of error and calculate the sample size based on that formula. However, since they really determine one another, one can start with sample size and confidence level then compute the margin of error. Determine any two, plug in to the formula, and get the third one.
Some other jargons regularly appearing in publication is the sampling technique. Different sampling techniques will not be discussed here, but we will comment that generally a certain sampling technique is applied in order to do one of these: increase accuracy, decrease bias, simplify, systemize, and ease sampling process, as well as having a better data breakdown by area, gender, age etc.
One other information that is very crucial to understand Quick Count result is percentage of incoming data, which is simply the ratio of data the survey institution that has been reported by their field officers to all the data from all polling stations they surveyed. It is important to read the Quick Count result with this ratio in mind as early reports may come from areas where a particular candidate is stronger. Trusting Quick Count result while the incoming data is still far from 100%, especially when the election is still going, is risky.
Despite all the risk of being misinformed, history showed how popular Quick Count results are for Indonesians. Therefore it is vital to be an informed reader. A quote widely attributed to Andrew Lang says, “An unsophisticated forecaster uses statistics as a drunken man uses lampposts; for support rather than for illumination”. One fitting addition to this quote may be “an unsophisticated reader reads statistics literally rather than rationally inclined to skepticism”.