Interfacing is a PITA

Josh Marshall at TPM has been having a series of very interesting outside experts write in on what they see as problems.  I think a few recent posts are hitting a highly probable failure point and an explanation as to why the state built and run exchanges are running better on average than the Federal exchange.

On Federal interagency interfacing efforts:

All the heavy lifting takes place on the back end, when the website passes your data to an extremely complex array of systems that span multiple agencies (like so many cooks in a kitchen). A central processing hub needs to get data from each of these systems to successfully serve a user and sign up for insurance. And if one of these systems — several of which are very old in IT terms– has a glitch and can’t complete the task, the entire operation fails for that user. Only if everything works perfectly, and the data gets passed back to the website, does the user have a good experience with

On Federal Exchange to Insurer interfacing:

 Given that 25 states (or more) refused to create their own exchanges, the federal government became responsible for creating all of them, in one application. That’s 25 exchanges each with multiple insurance providers, all of which likely have their own unique interface requirements, meaning that they need to receive information from in a format they can use, or can at least re-format, to work in their own enterprise architecture.

The first set of problems involve making the IRS computers talk to HHS computers which then talk to CMS computers which then talk to however many other entities that all have kludged systems working.  This is a common problem set no matter who built the Exchange front and intermediate layers.

The second comment has some useful insight.  Almost all insurers who process claims electronically use common (federally mandated) file formats and record keeping systems (for the geeks these are the 5010s, 834,  835, 836m 837, 838 etc) but there is definate variation between firms as to exactly how they define discretionary elements. 

State built exchanges have three significant advantages over the Federal Exchange.  The first is that there was political will to make things work so active problem solving has been an ongoing process.  Secondly,the number of unique data interfaces at the state level is far lower than the number of data interfaces that the Feds must manage.  Finally, most states that are building their own Exchanges are working with insurers where there was a pre-exisiting working relationship.  I know that we interface data daily with multiple state agencies.  They know how we send data, we know what they expect to see and vice versa.  Yes, there are new entrants to the market (co-ops etc) but most interfaces are building off of previous relationships.

50 replies
  1. 1
    Bill E Pilgrim says:

    I can’t imagine that there’s anything more to be said now that Chuck Todd diagnosed the problem. What more could anyone want?

  2. 2
    GHayduke (formerly lojasmo) says:

    Seeing as richard referenced the second post.


  3. 3
    WereBear says:

    I’m shocked at how many people (I’m looking at you, TV journalists) are acting like a huge amount of traffic that compromises a website is news.

  4. 4
    Omnes Omnibus says:

    @Bill E Pilgrim: I am not sure. Wouldn’t you want to get Jake Tapper’s take as well?

  5. 5
    chopper, interrupted says:

    there’s also the fact that red states that refused to have anything to do with the ACA are not exactly bending over backwards to interface their own data with upon request.

  6. 6
    srv says:

    Obama discovers the /. effect.

    I can tell you two states had front-end problems and apparently had never heard of loadrunner.

    So either clown-car or libertarian neckbeard saboteurs. You make the call.

  7. 7
    The Snarxist Formerly Known as Kryptik says:

    I’m honestly lost as far as the true significance of the issues here, and that’s because I’m admittedly ignorant as far as how complex and sprawling the work here needs to be, compared to how complex and sprawling it actually is currently, plus, you know, the traffic issues etc.

    I’m just frustrated to into apathy the way so many people (again, including erstwhile liberals) who have taken this as proof positive that Obamacare is the exact disaster the GOP said it would be and OMG Obama is the worst most spendy Devil-Satan Ever.

  8. 8
    Yatsuno says:

    The IRS’s main computer system is old. Like runs on COBOL old. I have two separate programs just on my computer that allow me to interact with it on a Windows environment and both get very fussy since the main system is so literal. Now spread that complexity across 25+ states. Yeah, and get your budget chopped by Congress constantly too. Is it any wonder this thing went boom?

  9. 9
    Gene108 says:

    Does anyone know if HHS ever asked and/or was given more money to get up and running?

    I know Sebelius was asking industry for help with certain parts of promotion and roll out because Congress refused to give adequate funding to HHS to fully implement and promote the ACA.

  10. 10
    srv says:

    Another thing, why is the IRS involved, I thought income pre-check wasn’t part of the current implementation. The IRS health credit should be a post-check thing like underreporter and EIC shit.

    The SS Admin owns SS numbers. IRS has to talk to it, not vice-versa.

  11. 11
    MomSense says:

    I am very frustrated that there is sooooo much hostility and scorn about the problems with when the Republicans have been preventing us from spending any money to upgrade our IT across all governmental agencies and departments. The same situation exists at the state level where we are trying to run departments with ancient equipment.

  12. 12
    Anniecat45 says:

    I’m an employee of a state government, and almost all of our major projects suffer from the same problem: the service provider bids low to get the contract, then can’t complete the work for that cost (and they often know or suspect this when they submit the bid); they then charges huge additional fees to fix the problems, which the state has to pay because it needs the system and it would cost vastly more to start all over again. I wonder if this is part of what afflicts

    No system is going to work if you don’t provide enough resources to make it work. I get so sick of people — I’m lookin’ at you, GOP — demanding cuts in government staffing for years and years then complaining because government doesn’t do every single little thing perfectly the first time.

    Next: private business systems fail, too, all the time, but they are considered “private” so all the problems are not out in front of God and everybody.

    And: has anyone, ever, had any computer system work well when it was first launched? the Apple service tech who sold me my iPad had trouble getting everything set up!

  13. 13
    Jim, Foolish Literalist says:

    @Gene108: I just heard on MSNBC that they got $1 billion to cover what was estimated to cost $5 B, I don’t have any links or more complete data, but it seems to me that this, if it hasn’t already been contradicted, is a pretty big deal

    One insurance industry source did say Tuesday that although the volume of enrollees remains low, the files coming over from [the Department of Health and Human Services] are now accurate. Some health plans have reported getting incorrect, garbled and duplicative files.

    My understanding is/was that this is/was a much bigger problem than the problems with the main website

    Read more:

  14. 14
    Another Holocene Human says:

    This might explain why is working so poorly in Florida. Scott decided to deregulate the new, ACA compliant healthcare plans that start 1/1/14.

    He was hoping the costs would jump up but they didn’t, really, from what I can see. However, it does make signing up complex. Just a giant fuck you to voters from the governor (again).

  15. 15
    ruemara says:

    I can speak to websites and political pressure. In our local municipality, getting the newest version of the website became a credo publishing priority because – upcoming races, natch. After a week of disaster with much missing, I sat in a departmental progress meeting hearing all that was wrong. I asked if the departments had approved a site architecture plan to confirm what should be in the website prior to take down and put up and if the existing site had been reviewed by each department prior to going live. The look of confusion and questions on what was a site architecture plan was amusing. This happens at all levels. And the “public/private partnership” co-operative government mode is a huge boondoggle/grant to connected contractors. Alas, my job is to deliver mail and I have no idea what I am talking about.

  16. 16


    Another thing, why is the IRS involved, I thought income pre-check wasn’t part of the current implementation.

    It may not be, but they’re involved with the implementation of subsidies. If the system takes you all the way through buying the insurance, it needs to be able to talk to the IRS to let them know that people are applying their subsidy to their monthly payments. If that step fails, they’ll probably need to revert the whole transaction.

  17. 17
    Another Botsplainer says:

    @MomSense: Yeah because you know Apple and Google and Amazon just works. Of course, it took them years to build their infrastructure and operations spending whatever needed to be spent. It didn’t happen in two years and wasn’t contracted to the low bidder. We may live in a world of technology, but I’ve seen some the dumbest stuff written about that I’ve ever seen on the Internet. Not to go off topic, but based on the ridiculous reporting on this subject maybe there should have been more skepticism surrounding Edward Snowden’s management PowerPoint dumps.

  18. 18
    Yatsuno says:

    @Roger Moore: SS gets income information before the IRS does anyway. But the IRS needs to know if someone has insurance coverage in order to determine if that person should have the penalty assessed as well. So IRS involvement is on several levels here.

  19. 19
    grrljock says:

    Thank you Richard for another informative and interesting post. I’m getting back to my initial interest in health policy and health informatics, so your posts have been really informative to me. E.g., I had no idea that almost all US government entities are forbidden to do cost-effectiveness or decide that certain drugs are too costly. I had thought that such craziness only applies to gun research.

    Back to this post, yes, it makes all the sense in the world why the state exchanges are more straightforward vs I have to agree with the person of (that Billmon linked to), that it’s actually a wonder that works at all.

  20. 20
    Walker says:

    Ah, data integration. Tons of theory research papers; little real progress.

  21. 21
    gussie says:

    So the feds didn’t establish separate sites for each state, along with a central site that simply brings you to the right one?

  22. 22
    The Red Pen says:

    I’m working on a Federal IT project and even the wingnuts in the dev lab aren’t ragging on this implementation. Most people are surprised it ever worked.

    One of my coworkers worked on a system that interfaced with the IRS.

    1. His system would call a web service at the IRS.
    2. This service call would result in a form being printed on a printer (on paper).
    3. An IRS gnome would take the form and process it using 19th-century technology.
    4. When finished, the gnome would key in some results.
    5. A “results” web service would communicate these back to the originating system.

    Most of the time, the delay between making the initial call and the results being available on the result call was about 4 days.

  23. 23
    roc says:

    @Yatsuno: Any wonder? It was the intended effect. They’re still running “Starve the beast”. Badmouth the system, sabotage the system, claim it as proof that your philosophy is true and correct.

  24. 24
    The Red Pen says:

    One more anecdote:

    Years ago, an intern from a local community college wrote a application to help with a common clerical task. It turned out to be so useful that everyone shared it and it became a fixture in the workflow. As you might imagine, it was quirky and crude, so naturally, someone suggested that they hire some pros to replace it.

    The DOD has spent — as far as my coworker can estimate — $50 million dollars on various efforts (including one by Oracle) to replace this utility and not a single one has succeeded.

  25. 25
    srv says:

    @Roger Moore: It just seems like this would be simpler to implement like a W-4 and if you screw up, the IRS comes after you in after you file.

    All the IRS processes are set up to catch you after the fact, not pre-crime.

  26. 26
    chopper, interrupted says:

    @Another Botsplainer:

    yeah, jon stewart was infuriating on that. “hey, works great!” yeah, dumbshit, it’s only been around 20 fucking years. i’m sure it was the bomb in the first week it went live.

  27. 27
    theturtlemoves says:

    As someone in the actual IT field who has had to deal with complex data integration, but nothing like the scale we’re talking about here, I am inches away from punching random strangers every time I watch the news. It isn’t just a “Web site”. The Web site is the presentation layer of a system that has hideously complex data and application layers. So, no, you can’t just throw more servers at it and no, it isn’t comparable to a bunch of hits on some freaking eCommerce site that has maybe a few databases on the back end that were probably actually designed to work together. The comment that killed me Monday while watching CNN at the gym was John McCain saying we should get Air Force one to pick up 200 geniuses in Silicon Valley to fix everything. I don’t care how freaking good you are, it is going to take you a couple weeks to get up to speed on the data, the code, the environments, etc. So, why don’t we just let those poor bastards who are likely sitting under their desks weeping right now (been there) fix the bugs without getting advice from Chucky Fucking Todd and Grandpa Walnuts.

  28. 28
    StringOnAStick says:

    My husband is a senior IT guy (as liberal as me – they aren’t all wingnuts!); he was hired 14 years ago for his current private sector job. He, and dozens others over the years, were hired to migrate the firm from their old, kluge-filled systems to the new spiffy one they were tasked with creating because the old systems simply cost too much to maintain, could no longer keep up with the evolving business rules, and were in COBOL for gawd’s sake.

    The company offered to retrain all the COBOL people into the latest and greatest for programmers so they could remain employed at the firm, and since the old BOLer’s had seen numerous consulting IT firms fail at the overall task 95% of them told management to FOAD and didn’t want to or see a reason to ruffle their insular world. 14 years later and that 95% are all on the streets, but, and this is a HUGE but, the old systems still live and must be interfaced with, thus making the current system even more complex, more of a kluge-fest, and most of all, a guarantee of permanent employment for my husband. So, sort of winning?

    Needless to say, my husband thinks all the wailing over the healthcare websites is more than a little disingenuous, if not completely insane.

  29. 29
    Jim, Foolish Literalist says:

    @theturtlemoves: The comment that killed me Monday while watching CNN at the gym was John McCain saying we should get Air Force one to pick up 200 geniuses in Silicon Valley to fix everything

    Even a solid Luddite like myself had to laugh at the amount of wrong the old coot had to pack in to that sentence, to say nothing of the unremarked fact that he was essentially affirming that Obamacare should be preserved and improved, not repealed and replaced.

  30. 30
    Gene108 says:

    @Jim, Foolish Literalist:


    So far what I have heard reported is the website cost $300 to $400 million and is a total failure.

    There’s no indication if $400 mil was enough or if more money is needed because 36 states refused to set up exchanges.

    Very frustrating how the reporting has been framed, though not unexpected.

  31. 31
    Steeplejack says:


    The comment that killed me Monday while watching CNN at the gym was John McCain saying we should get Air Force One to pick up 200 geniuses in Silicon Valley to fix everything.

    Yes, throw manpower at it. Genius manpower! “Why, if we put nine women on that pregnancy we’d have a baby in one month!”

    This is why Frederick P. Brooks’s The Mythical Man-Month remains a classic.

  32. 32
    wenchacha says:

    However, I did not get my pony. Therefore, Obama may as well be Cheney, right?

  33. 33
    pseudonymous in nc says:

    @chopper, interrupted:

    jon stewart was infuriating on that. “hey, works great!” yeah, dumbshit, it’s only been around 20 fucking years.

    And when AWS has a big outage, a big chunk of all the cool new sites that run on its infrastructure go poof temporarily.

    But again, the issues with aren’t technology problems. The user-facing tech (and even the backend tech) are symptoms of procurement problems, and some of those procurement problems go all the way back to shitty legacy systems. The cue from is to start changing the procurement model.

  34. 34
    ericblair says:


    . The comment that killed me Monday while watching CNN at the gym was John McCain saying we should get Air Force one to pick up 200 geniuses in Silicon Valley to fix everything. I don’t care how freaking good you are, it is going to take you a couple weeks to get up to speed on the data, the code, the environments, etc.

    That, and it’s not exactly geniuses that are needed at this point. If the documentation on how to interface to the Missouri department of revenue individual return database is one page of out-of-date database fields and an incorrect POC email, Albert fucking Einstein isn’t going to help you fix that.

    There’s also the Silicon Valley map-is-the-territory mindset that if you’ve got a nice flashy app for whatever ails you, that’s the solution. For Obamacare, is just the way to get the product, not the product itself, and it will get fixed bit by bit by your normal workaday developers figuring it out with other normal workaday developers if you give them the fucking time and resources.

  35. 35
    ericblair says:

    @Jim, Foolish Literalist:

    Even a solid Luddite like myself had to laugh at the amount of wrong the old coot had to pack in to that sentence, to say nothing of the unremarked fact that he was essentially affirming that Obamacare should be preserved and improved, not repealed and replaced.

    That’s what happens when you try to pivot from “BURN THE MOFO DOWN” to “the fire alarms don’t work right!” in about ten minutes. I don’t think most people know the term “concern troll”, but do understand the concept.

  36. 36
    Another Botsplainer says:

    In cases like this it is generally infrastructure tuning parameters and interface tuning parameters that fix problems. The other big culprit is poorly written SQL by developers. It takes TIME to fix these types of problems, super geniuses who write stupid worthless shit like Facebook are of no help.

  37. 37
    Heliopause says:

    Yes, we hired a team of technocrats to reform health care. The technocrats told us that the most sensible solution is not politically possible so we said, fine, we’ll settle for the rube goldberg solution you are proposing instead. As even people who are not technocrats with Ivy League educations know, rube goldberg solutions are prone to entirely predictable problems such as those outlined above, but there we are. So, time for the technocrats to fix the problems, leaving the recitation of excuses to the blog comment sections.

  38. 38
    agrippa says:

    Tome, a couple of things are clear: The GOP will try to exploit this to the maximum extent.
    Second, most of the ‘news media’ have no knowledge of the details; have no interest in learning.
    Very few will make any effort to learn and inform.
    Third, when it is fixed and running fine, the media will be completely silent on the matter.

  39. 39
    Another Botsplainer says:

    @agrippa: Kind of like the IRS “scandal”.

  40. 40
    bemused senior says:

    At last, an informed article that lays out the back end problems in detail. Yay Sarah Kliff for journalism.

  41. 41
    Richard Mayhew says:

    @bemused senior: 834s are another issue, although bad 834s happen all the time in group insurance.

  42. 42
    Lynn Dee says:

    Yep. This is the truth. All the status meetings, working groups, sub-working groups, sub-sub-working groups, specification documents, design reviews, code walkthroughs, formal and informal testing, configuration management and now, ongoing exchanges of email, will prevent interfacing from being a never-ending source of trouble.

  43. 43
    bemused senior says:

    I found the data flow diagram in Kliff’s article informative. Obviously the EDI exchange is crucial, but you can see the other interfaces that have to work by examining the DFD.

  44. 44
    BruceJ says:

    @Jim, Foolish Literalist: And while we’re at it, lets get those Sunnis and Shiites into a room and hash this thing out.

    More and more the whole thing is starting to resemble This

  45. 45
    geg6 says:

    I am not an IT person. I also know next to nothing about insurance or insurers and their systems. But I do think there is a federal online system out there that is similar to, in that it has to interface with separate systems in every state and with every school that gets Title IV funding, each of which has their own separate systems. It also has to be able to do data checks with multiple federal departments (the IRS, INS, DoJ, DoD, VA, SSA). That would be the FAFSA. It has to be able to send information to each state’s higher education department and to every school (there can be up to 10) the student lists on the form. I don’t know enough to know if my example is valid, but my experience, from the very first year they put the FAFSA on the web until today, is that it was a mess when they started the online version of the FAFSA (can’t remember exactly, but I think it was during the Clinton administration), got marginally better every year since and has gotten so good that the FAFSA and IRS website are now so tightly linked that people can download their 1040s directly onto the FAFSA. Every time the Higher Education Act has to be reauthorized, the FAFSA has to change and so do all the state agencies and higher ed institutions. It’s never seamless and it’s sometimes a huge clusterfuck. But nonetheless, we all manage to get students their aid. I can’t believe this will be any different.

  46. 46
    agrippa says:

    @Another Botsplainer:

    After an indecent interval, the ‘issue’ will disappear.
    Both the GOP and the ‘news media’ will lose interest.

  47. 47
    spaps says:

    @Richard Mayhew: I work in IT for a health insurance company and knew immediately upon reading the multiple spouse issue that it was probably incorrectly filled INS02 elements for dependents.

    My company has decided to delay participating in the exchange until 2015, but your comment on the common file formats is dead on for existing interfacing, especially regarding the 834 format. The 835 & 837 seem to be more strictly defined, but it still comes down to what is listed in the companion guides exchanged between trading partners of what are allowed and acceptable values in the segments and elements.

    This stuff. Is. Complex. It doesn’t surprise me that there are issues out of the gate regarding data being filled incorrectly, especially in cases of states that refused to set up exchanges for the reasons you noted above. I worked on converting our inbound 834 process from 4010 to 5010, and the state was a godsend for testing. Being able to receive parallel 4010 & 5010 files for CHIP, Medicaid, and County Health Plans helped me clear up errors in the 5010 834 process in my company before it went live. I’m curious what level of testing has occurred in the case of the federal exchanges with the insurers in these states. Based on the article bemused senior posted, I’d imagine there was little if any for 834 files.

    I now work mainly with EDI claims & remittance advice files (837 & 835 respectively), and have since the beginning of the year. I’ve learned more than I ever cared to know, and still have more to learn. ICD-10 and CORE operating rules are the hot items at the moment as I’m sure you’re aware.

    I’ve enjoyed your posts so far because they unveil just how complex healthcare and insurance really are. Keep it up!

  48. 48
    adhocheretic says:

    I work in state government IT on software sitting in the middle of 6-12 other pieces of ranging anywhere from a year old to being more than old enough to have a drink.

    I’m amazed the front-end site is live, never mind any of the back end processing.

  49. 49
    Doofus says:

    I am still surprised they thought they could get all of the back end transactions and queries done in real enough time for one web transaction. Pay/pursue is much easier than pursue/pay. And whoever talks about dups being an issue for insurers doesn’t know what they are talking about. Not an issue at all.

  50. 50

    […] Richard Mayhew, there are some articles by tech experts at TPM discussing the federal insurance exchange website […]

Trackbacks & Pingbacks

  1. […] Richard Mayhew, there are some articles by tech experts at TPM discussing the federal insurance exchange website […]

Comments are closed.