Harold Pollack had a good post in 2012 on the math of screening and the importance of a priori observations:
since a test with greater than 99% sensitivity and specificity can still yield some pretty screwy results when it is deployed within the wrong population.
Suppose we use the above test in a pretty high-risk population in which the prevalence of unknown HIV infection is about 3%. That’s really high….In this population, the rapid test really seems to work well. It has high positive predictive value, and it has incredibly high NPV, too. Incidentally, you can now see why the commonsense term ”accurate” is a pretty sloppy way to discuss screening tests. Depending on the context and underlying prevalence, that word could denote any one of these four things: sensitivity, specificity, PPV, or NPV.
In any event, what happens if we apply the same test within a lower-risk population, such as the population of young sexually active straight couples likely to be a key market for this test. Within this population, HIV prevalence is more like 0.2% rather than 3%….
But the test doesn’t seem so “accurate” anymore. The positive predictive value drops to 20/40, which Austin tells me equals 50%. That’s a serious problem. Half of the positive test results will be false positives.
Austin Frakt at Academy Health digs into the same wellness effectiveness study that I mentioned a couple of months ago. As a refresher, Pepsi had a general lifestyle wellness program (lose some weight, eat more fruit etc) and a chronic disease management program that they allowed for a systemic evaluation. The results were aggregated as successful but the breakdown was that the disease management component drove all cost savings and health improvements.
The takeaway is a familiar story. When narrowly targeted wellness programs, like many other health interventions, can be beneficial–even cost saving (a rarity among health interventions). But, when more broadly implemented, they often are not. A focus on workers and dependents with specific diseases makes eminent sense. Not only are they the sickest–and in that sense deserving of greater focus–they are the most expensive to insure, offering a far greater opportunity for savings from disease and lifestyle management than a typical insured.
Prevention works when there is an a priori belief of elevated risk in a population. Screenings are the same thing. General screenings for unusual diseases have a hard time covering costs if the screenings cost more than a couple of dollars because too many people have to be screened to detect one case that otherwise would not have been detected. There are exceptions; most vaccinations are amazingly cost effective, clean water, bleach, and steam are amazingly cost effective but there are very few very low hanging fruit that a broad population can benefit from.
Belafon
There is so much ignorance we could cure in this country if we required a semester in school on Bayes theorem. “An entire semester on one equation?” Yes, because while the equation is small, it is counter-intuitive to a many lessons we’ve learned throughout evolution.
maximiliano furtive, formerly known as dr. bloor
Balloon Juice: come for the trolling, stay for the Base Rates representin’.
WereBear
Our esteemed bloghost has now learned firsthand how many of his relatively minor, but highly draining, problems could be solved with food. And I believe my own attention to what I eat will stave off diabetes, which killed my dad last summer.
Intervention CAN work. It just has to be the right kind.
Richard Mayhew
@WereBear: But it is targeted intervention — JC had pre-exisiting risk factors and made relevant modifications to his life. A broadbased intervention for everyone at Balloon Juice would definately catch JC but it would also catch the current vegetarians, it would catch the people who are already taking care of themselves.
I agree, interventions can work, and they do work, but targetting narrow populations with risk factors is the way that they can be both effective and efficient.
Richard Mayhew
@maximiliano furtive, formerly known as dr. bloor: Just for you — the reason for my low output this week has been that I and most of my group has been on a clean-up detail from a screw up that was 100% on the insurance company end — the only doctor involvement was an office calling us to say something looked funny :)
hoodie
Did you purposely time this post after the one on Brother’s Keeper? The interesting thing we see is general wellness programs being pushed even though, as you indicate, they’re of dubious value. Maybe that’s because we are so intent on assuming an illusion of equality of opportunity and politically biased against anything that might involve “special treatment” of a particular people by government programs, even if that special treatment will end up costing all of less in the long run. You know, kind of like universal
preschool is cheaper than cops and prisons.
maximiliano furtive, formerly known as dr. bloor
@Richard Mayhew: HAHAHAHAHAHAH! I roolz the internets.
Cermet
There are a few HUGE pieces of hanging fruit to catch!
First, require a prescription to obtain the most deadly over-the-counter drug bar none: Tylenol. Besides causing thousand’s of liver failures (most will recover) a year and while many hundreds/year will require a liver transplant. Then there is the birth defects and ADHD increase that this drug has been proven to cause in pregnant woman.
Statins are not all they are cracked up to be – fantastic IF you have suffered a heart attack; very iffy if you haven’t. In the population that hasn’t suffered a heart attack, it will save 1 in every 60 people (so far, that is great) but about 1 person in every 100 will get ALS that normally would not!!! That is far, far too dangerous – this excludes the kidney failure issues and liver failure (both these complications will occur in 5% of all patients but most will stop before real harm; however, for some, esp. kidney failure, these will die) issues.
The addition of Vitamin D (it isn’t a vitamin) to everyone’s diet would reduce many cancers and reduce general illness across the population by huge numbers if the FDA increase minimum dose levels to at least a 1000 IU a day (2000 far better.)
WereBear
@Richard Mayhew: Ah, now I get it… I would be “targeted” because of my father’s illness, for diabetes.
Cassidy
@Cermet: If people followed the instructions, Tylenol toxicity wouldn’t be an issue.
catclub
@hoodie: ” You know, kind of like universal
preschool is cheaper than cops and prisons. ”
So in that case, the universal approach is better than the specific one.
grillo
@maximiliano furtive, formerly known as dr. bloor: You misspelled ‘teh’.
hoodie
@catclub: Actually, no. It targets young people. Universal pre-k is good for both affluent and disadavantaged kids.
Mnemosyne
As I’ve mentioned before, we have a “wellness program” at the Giant Evil Corporation, but it many ways its purpose seems to be to identify and steer people into more targeted disease management programs. I don’t know how successful it’s been at that, but it does seem to make sense for a large company to try and pinpoint the employees who would benefit from disease management.
But it’s also what hoodie said: the programs and their benefits have to be extended to all employees because otherwise employees get pissed off that some people are getting “extra stuff” that they aren’t.
FlyingToaster
@hoodie: I can attest to that. My daughter’s kindergarten class ALL had pre-K, either at her school or at a dedicated preschool. And those kids are both better adjusted and further along than my K back in the 1960’s.
If it were left to me we’d have multiple preschools in every zip code; my daughter went to an arts-based school; the girl across the street went to one on a farm, and the boy up the road went to the town’s.
The biggest problem we see locally is how to pay for this; our town’s public schools are in a space crunch (they sold 2 schools, tore one down to build a police station, and converted a school to the pre-K/SpecialEd/admin building), and don’t have enough seats for all resident kids for either Pre-K or Kindergarten. While a lot of residents have gone private (like us), it’s not an option for the majority, and with 1st and 2nd grade classes averaging 30 per (there are 179 2nd graders and 187 first graders, across 3 schools), it’s getting VERY tight.
JoyfulA
I worry about false positives on drug tests, now that a ridiculous number of jobs and programs require “screening.” I keep thinking that a lot of people must be tripped up, falsely accused, etc., but I never see any news along this line.
tk
@maximiliano furtive, formerly known as dr. bloor:
+1, as the kids say
Joel
That’s a bit of sleight of hand; what you’re saying is there’s a 1% chance of error in the absolute sense. It’s hard to imagine a screening test, which are usually antibody (ELISA) based, repeatedly generating false positives for someone who is not infected by HIV. False negatives are more of an issue with the “rapid” screen.
carolannie1949
@Cassidy: A lot of people take acetaminophen without knowing it: in cough meds, cold meds, and then take an extra dose for their aches and pains. Usually because they are in pain and want to resolve the pain, AND the cold or flu symptoms. There are a lot of OTC meds that should not have acetaminophen in them at all, but appeal to people who don’t know any better. AND WHY SHOULD THEY? They trust the drug companies! They are told again and again that Big Pharma is the source of all good drugs and the FDA has our best interests at heart.
Mnemosyne
@JoyfulA:
Frankly, most of the jobs that require screening these days are minimum wage crap jobs, and who cares about those people’s civil rights anyway?
/snark
I actually hardly ever hear about mandatory workplace drug testing, but I’m in the entertainment industry where it would be pretty useless. I think the employees at the company my husband works for get drug tested, but he works for a home infusion pharmacy, so it’s probably a good idea to screen out current opiate addicts ahead of time.
low-tech cyclist
I don’t have anything useful to add here, but just wanted to say, Richard, how much I appreciate your posts. I for one like having a little bit more background on how this stuff all works than we’re likely to get from CNN. Thanks for fighting our ignorance here.