Last spring I was very suspicious of the RAND study that claimed a good deal of the net gain in insured individuals was from people who picked up employer sponsored insurance (ESI). I thought it could have been a sampling problem as the subset was very small with a large margin of error in both percentage and absolute terms. Others thought there was a methodological flaw :
This poll is unusual in that it interviewed the same people twice, first in September and then again in March. This allows it to see how many of the sample actually change from one type of cover to another. But its weakness is that inevitably somewhere between 5 and 10% of the people you interviewed in the first round can’t be contacted again. Some may refuse to answer more questions, but more likely the phone number has been disconnected. RAND reports that 2641 people were interviewed in September but only 2,425 of these (91.8%) were interviewed in March. So RAND went ahead with those who answered both surveys, assuming that were similar to the whole population.
The RAND study’s headline results were similar to everyone else’s on changes/improvements to the uninsured rate from early last fall to after the end of the open enrollment period. It was just their subdivision of how people gained insurance was wrong. They were the only ones to attribute a large proportion of the coverage increases to ESI taking off, and thus a much smaller proportion of the gain to either Medicaid expansion, woodworkers or Exchange policies. It did not pass the immediate smell test.
However, there is a very interesting nugget from Walmart that supports the Rand theory:
The nation’s largest retailer said Thursday it now expects to spend about $500 million for U.S. health-care costs this year, up from a previous estimate of $330 million. A larger number of employees seeking coverage as well as other medical costs that are rising contributed the company’s higher expenses and its decision to trim its earnings forecast for the year.
Rand’s theory was that the threat of a mandate payment got a lot of people who were making the marginal decision to not get coverage at work to get coverage from work. Walmart seeing their health insurance costs increase by 50% in a year where health insurance inflation is between 6% to 9% strongly suggests most of that added cost is being driven by more people getting covered by Walmart.
Interesting….
Belafon
I am going to lose no sleep over Wal-Mart being forced to cover its employees.
Betty Cracker
Boo-fucking-hoo, welfare queens. It’s about time you paid up and quit foisting the cost of Medicaid coverage for your impoverished workforce off on us taxpayers.
jl
‘ So RAND went ahead with those who answered both surveys, assuming that were similar to the whole population. ‘
‘ Others thought there was a methodological flaw ‘: and these people would be 100 % correct. Just blowing off that kind of loss to followup is bad practice, and I challenge anyone to produce sound expert statistical advice to contrary.
There are established methods for doing sensitivity analysis to find out how likely the attrition made a difference in the results, and put bounds how big or small that difference might be. Or you use whatever information you have on those who dropped out to impute a range or likely results if they had not left the sample. And there are a dozen ways of doing this, from plugging in some summary statistics to very sophisticated predictive data mining techniques, that can be validated to some extent.
This is pretty basic stuff you get in intermediate statistics, and I’m surprised a place like Rand would let that slide.
The Walmart claims are interesting.
But, my big question to RM is: So what? My understanding of the post and the link is that some people looked at exchange option and its net cost, penalty for remaining uninsured, and looked at employer policy, and decided the last was the best deal for them. So?
Mike in NC
Sounds like the Walton clan might be buying one less Rolls-Royce.
Ronnie Pudding
Getting 92% of participants to complete the follow up survey sounds excellent. I work in market research (consumer products based) and we’d kill for that.
Villago Delenda Est
@Betty Cracker: Keeping in mind that this sum of money the Walton vampires find in the couch cushions.
Violet
Of course it had nothing to do with better health insurance options at work due to the requirements of the ACA. Or being eligible to get health insurance at work due to the requirements of the ACA. Nothing like that at all.
Tissue Thin Pseudonym (JMN)
@jl:
So basic, in fact, that one ought to be suspicious of a claim that RAND made such an error. In this case, the people Richard is citing about the error just didn’t read the actual report properly. Here is their claim:
Here is what the RAND report actually says:
That footnote at the end is kind of important:
So, the claim that the survey had an almost 10% attrition rate that RAND just handwaved away is, in fact, false. Charles Gaba and, by extension, Richard Mayhew have just flat out gotten it wrong. 2,614 is the number of people interviewed in March and the difference between that and the figure of 2,425 used in the sample came about because RAND thought that they specifically didn’t belong.
The largest group of this actually went the other way: they were people who were in the pool that RAND asks questions of monthly who did not respond to the first survey in this series, the one in September, 2013 and thus had no data to compare them to. Methodologically this is a different phenomenon than people who were in the first sample but not in the second and is closer to those who were non-respondents to begin with and thus never made it into the pool.
The criticism of this study is also a case of looking only at the point estimates without mentioning the margin of error. In this case, the margin of error is pretty wide given the sample size, which makes me suspect that RAND did do all of the things you discuss in dealing with the different survey respondents between the two months. And you can’t blame RAND for this omission, because they mention the margin of error in their press release, albeit only the one for the change in uninsured population as a whole.
It appears that Gaba based his criticism of the study based only upon the press release and even an incomplete reading of that.
jl
@Mike in NC: Cry not for the Waltons. Economic theory and statistical evidence strongly suggest a large chunk, perhaps all, the additional health care insurance costs will come out of Walmart employee wages within a few years. So, this is a short-run nick in their profits that will heal quickly.
Though, now that more people can get a better substitute for employee provided health insurance outside of work, at what point does the ACA produce an increase in many industries’ cost curves, so that the fraction of the premium actually paid by the employee even in long run decreases and that by the employer increases? Does RM have any ideas on that?
Tissue Thin Pseudonym (JMN)
@Violet: Or, maybe the person summarizing RAND once again did a lousy job of it. Here’s the relevant bit from the report:
What RAND said is a lot more equivocal than what Richard claimed they said, as that falls well short of being their theory. It’s more like they were tossing out ideas. I don’t think it deserves the kind of scorn you treat it with.
jl
@Tissue Thin Pseudonym (JMN): thanks. I downloaded the pdf report and the data appear to be a continuing survey with a rotating panel (that is, the survey design intentionally consists overlapping cohorts or randomly selected people who stay in the survey for a certain period of time and they are replaced by another randomly sampled cohort).
So, yes, you are right, you have to spend some time counting up what is true loss to follow-up and incomplete response versus just rotation of cohorts in and out of the sample.
There results on last page indicate marginal statistical significance. Maybe the report for the bigger sample they say is coming will be more informative about what they did.
low-tech cyclist
Coupla thoughts here:
1) 91.8% is a damned good repeat response rate, especially after six months. “Only,” my ass.
2) Where I work, there’s a thing we do called a ‘nonresponse bias analysis’ to try to see if the survey results were skewed one way or the other by who responded and who didn’t. When you’re coming back to people you’ve already interviewed, you’ve got more tools for such an analysis, because you’ve got all the data (including demographics) on your nonresponders to the subsequent round that you collected in the earlier rounds, to compare with what your repeat responders look like.
So do we know whether or not RAND did such an analysis, or did they just ‘assume’ that the 8% of the sample they lost didn’t make a difference?
Matt
Another facet I wonder about is a consequence of the ACA banning some of the “insurance” products that are offered by large, low-wage employers. I remember one of them from years ago; had a huge deductible, only paid something like 70% for hospitalization, and had a lifetime cap in the $200k range – in other words, a “pay for this every 2 weeks and then still go bankrupt when you get sick” plan. IIRC, garbage plans like that are no longer salable. So I could see more Wal-Mart employees signing up for coverage *now* because they’ll actually get a real plan for their money…