Unfucking: Much More Difficult than Fucking

No more predictable lede than this could have been written about the failure of the insurance exchange software (emphasis mine):

The technical problems that have hampered enrollment in the online health insurance exchanges resulted from the failure of a major software component, designed by private contractors, that crashed under the weight of millions of users last week, federal officials said Monday.

Our friends at Demos have done a little bit of research on the whole government contractor grift, and here’s what they’ve found:

In this brief, we extrapolate from the GAO study to estimate that the federal government is spending an estimated $20.8 to $23.9 billion a year to pay private contractors for the compensation of top executives. $6.97 to $7.65 billion in taxpayer dollars is spent annually on pay that exceeds the U.S. Vice President’s salary of $230,700 a year.

In a nutshell, here are my educated guesses of why exchange software is failing:

  1. The massive overhead involved in government contracting (not just C-level salaries, but also the waste involved in the sales process) meant that percentage of the budget for exchanges devoted to actual work and workers wasn’t sufficient for the development and testing the sites needed.
  2. Government outsourcing culture leads itself to lack of accountability on the part of government employees and lies on the part of contractors. Lack of accountability comes from the contractor, not the contract manager, getting the blame for failure. Lies come from the contractor saying anything to win the contract, knowing that failure just means more money will be thrown at the problem.

But those are just my guesses. Maybe some of you who have been involved in contracted government software have a few other ideas.

Share On Facebook
Share On Twitter
Share On Google Plus
Share On Pinterest
Share On Reddit






141 replies
  1. 1
    aimai says:

    I know someone who does some of this kind of work for the government.The salaries are very, very, strictly controlled and calculated in a very complicated way. The CEO of his company takes, technically, a very low salary to keep the demands of the employees for comperable pay low, and to keep the overhead cost low (the cost which they can charge the government). This leads to underbidding in the sense that it can be quite a bit more time consuming to produce X than you bid, in order to make your bid competitive with that of other people. The end result is uncharged time coming out of the hides of some of the lower level contractors in order to fulfill the bid.

  2. 2
    Cervantes says:

    Was there no way to blame Apple or its iPads for this failure?

  3. 3
    Punchy says:

    Well….the title of this post is quite NSFW.

    Lies come from the contractor saying anything to win the contract, knowing that failure just means more money will be thrown at the problem.

    This is it. Failure is almost a desired outcome, b/c the govt will just throw even more money at it. If it succeeds right at the start, that contract is complete and the COO doesn’t get his new boat. In a perfect world, the developer(s) dont get paid for such shit work, but that’s not how it works.

  4. 4
    russell says:

    I think your (2) captures it pretty well.

  5. 5
    Another Botsplainer says:

    If the taxpayers are paying a GS13 programmer about 100K a year, they are paying 3x to 4x for a contractor to do the exact same work. But it’s all good because the contractor is not a lazy government freeloader.

  6. 6
    russell says:

    I think your (2) captures it pretty well.

  7. 7
    russell says:

    I think your (2) captures it pretty well.

  8. 8
    Another Botsplainer says:

    @Another Botsplainer: I should mention that the contractor is not making the 3x or 4x money the company is. In many cases the contractor makes the same or less the the government employee.

  9. 9
    joes527 says:

    As someone who has spent his entire career in the private sector, I always get a chuckle when folks suggest that fuck-ups and waste are somehow unique to the public sector.

  10. 10
    OGLiberal says:

    The site is horribly bad. I imagine they’re trying to fix it because I’ve been getting a “system down, try after 10am Eastern” for hours. Shocked they didn’t do the necessary load/bug testing before rolling this out. Don’t know who is at fault but heads should roll – this is the President’s/government’s signature accomplishment of the past 10-years. The shit should work. And I’m not talking about it being a little slow…it just plain don’t work. My guess is that the wingnuts aren’t making a big deal about it because they’d rather it not get fixed any time soon.

  11. 11
    OGLiberal says:

    The site is horribly bad. I imagine they’re trying to fix it because I’ve been getting a “system down, try after 10am Eastern” for hours. Shocked they didn’t do the necessary load/bug testing before rolling this out. Don’t know who is at fault but heads should roll – this is the President’s/government’s signature accomplishment of the past 10-years. The shit should work. And I’m not talking about it being a little slow…it just plain don’t work. My guess is that the wingnuts aren’t making a big deal about it because they’d rather it not get fixed any time soon.

  12. 12
    Emma says:

    Not to be contrarian but (3) many, many more people than expected tried to access the site? It wouldn’t be the first time in software history that that has happened. As someone whose only experience of public sector was a one and a half year stint at a public library, I can attest that software/hardware screwups are the norm, not the exception.

  13. 13
    TAPX486 says:

    My first job was with a defense contractor in the late 60’s. This was before the outsourcing mania took over but the culture was the same. The contractor would promise, for a dirt cheap price, to build the starship Enterprise. Ten years and billions over budget they would deliver the a replica of the Wright Brothers bi-plane. If anyone thinks that has changed then exhibit A is the F35 joint strike fighter. It is badly over budget with hundreds of defects and years late. It was meant to be a plane to be used by all of the airborne services. Unfortunately the tail hook, essential for a carrier landing. doesn’t fit. OOOOPS. But not to worry for another 20 billion dollars the contractor will think of something. I’m not sure if it’s the F-35 or the F-22 (famous for asphyxiating its pilots) but it can’t communicate with other NATO aircraft w/o losing it’s very very expensive stealthy capability. Now maybe this doesn’t hurt the long range and solitary B2 but in a jet fighter designed to work in tandem with other aircraft it is a bit of a problem.

  14. 14
    MattF says:

    Well, there’s a huge literature about how and why software projects fail– and it happens to both privately and publicly financed projects. I’d guess that the existence of political actors who are actively promoting failure probably doesn’t help.

  15. 15
    Tommy says:

    My first job out of college (back in the early 90s) was working at a market research firm that tracked federal procurements. One department I followed was the Navy. They had a billion dollar project to write a custom email program, cause I guess Outlook wasn’t good enough for them. One of the best things Gore did as VP was to do a lot of procurement reform, where you had to use COTS (commercial off the shelf) products. Contractors had to show pass performance when they would bid.

    I do web sites for a living and I know I can develop a web site that would do what healthcare.gov does with COTS products.

  16. 16
    OGLiberal says:

    @Emma: I think they should have planned for more, not less…always expect the worst. (best?) But it’s also taken way too long to address the issue. Should have been fixed in a day or two….a week later and the site is down…not just slow…down. And it’s been down for hours. And this is planned downtime. You should be down for, at most, 2 hours, to apply a patch. I want nothing more than for this to succeed but this is pretty sad. That said, it’s been a week. In the bigger scheme of things, not that long.

  17. 17
    mericafukyea says:

    LOL…HA..and all that. Thanks for the morning laugh muckymux. This coming from a guy who can’t even manage to deal with the Ball Juice wordpress website. Now suddenly you are an IT expert on scaling large scale custom dynamic websites.

    Always a hoot coming to this site and reading ball juicers talk out of their ass about things they know dick squat about.

  18. 18
    WereBear says:

    @Emma: A previous thread quoted the software engineer as saying they got five times the expected traffic. When the traffic drops, everything works as it is supposed to.

  19. 19
    Tommy says:

    @OGLiberal: My father was over the first of last week watching me hit reload, reload trying to set-up my account. He was like “dude don’t you do Web sites for a living? Should this happen?”

    I am like nope, this is like a 1995 problem, not a 2013 problem. I could pick up the phone and call like five different hosting companies (that is just the five I know) that could have handled the traffic.

  20. 20
    Kay says:

    @Emma:

    But you have to register to get in at all.
    They could have taken care of 90% of the people by just letting them look at what’s available. It’s a big purchase for people. They’re asking for advice, thinking about it, etc.
    I don’t think their “average customer” is a person with an Amazon account. How can they not know that?
    My biggest frustration over the whole health care debate has been no one seems to have any insight into who are the uninsured.
    The people here who will buy this have purchased ONE insurance product, and it’s state minimum car insurance. They don’t read the terms of THAT contract and they have no idea what they’re buying, but this is bigger for them.
    Why put them thru this initial bottleneck right at the start? Just let everyone in and then take purchasers thru verification, etc.

  21. 21
    joes527 says:

    @Another Botsplainer: 3x to 4x is extreme, but comparing a salary to a contract price is useless. You need to compare the fully burdened rate to the contract price. It still doesn’t come out to 3x, but it probably makes the delta less dramatic.

  22. 22
    Botsplainer says:

    keeping in mind that I know very little about the architecture or load, but how does the extent of the use of the healthcare site compare to:

    Amazon
    USPS
    PACER/ECF
    UPS
    IRS

  23. 23
    Kay says:

    @WereBear:

    I wish he’d stop saying that.

    If you use it, it doesn’t seem to be true. I didn’t have long wait times. It’s complicated, it’s an actual identity verification process, like a credit reporting site, and that part takes about as long as a commercial site, but it doesn’t work the same way twice. One time the error is “page not found” the next time the error is one doesn’t get the promised email, the third time it works fine. Is that volume? I don’t think so.

  24. 24
    becca says:

    Privatize, de-regulate, stabilize.

    That is the Washington Consensus. We should all brush up on the history of this godawful doctrine.

    Privatization is about creating private wealth at the public’s expense. Selling off the common good to like-minded sociopaths is part of the plan.
    Hamstringing democracy is the goal.

  25. 25
    gene108 says:

    OT: Called my Congressman’s office, Jon Runyan (R-NJ3), and asked if he’d vote for the Senate’s CR. Secretary said he’s in favor of a clean CR.

    I think if the CR is brought to a vote, it will pass, with Democrats and a few Republican votes.

  26. 26
    TAPX486 says:

    @Tommy: I’m not sure if they are just saying this to deflect blame but it does sound plausible. It isn’t just the web platform that is the problem. They had to design for the laws and regulations of 37 states, some of which were not co-operating, the policies of heaven knows how many insurance companies and then have a front end that could translate all of insurance gobbly-gook into terms that non-experts could understand Once that was done they then had to be able to translate it all into the languages non-English speakers use in this country.

    I’m not saying they didn’t screw up but there are an awful lot of moving parts. And let’s not forget the roll-out of Medicare Part D – the bush drug plan – crashed and burned in 2004-2006.

  27. 27
    amk says:

    shit happens in software, dood. vista, windows 8, … some site called bj.

    software is an equal opportunity fuck-up source. It doesn’t care whether it’s gobinment or private.

    Only sites that work pitch perfect are pr0n ones. That ought to tell ya something.

  28. 28
    Lee says:

    I’ve always been able to hit the healthcare site and play around a bit.

    Who is getting these errors? Is it regional? I’m in Texas

  29. 29
    Botsplainer says:

    @TAPX486:

    The F22 is the asphyxiator.

    The F111 was the real catastrophe. It ate a shit ton of money, was inadequate for its purpose since before it even left the drawing board, and they built a bunch of the fuckers anyway to drop mostly dumb ordinance in Vietnam. They accomplished more on a tactical basis with cheap old AC-37s.

  30. 30
    OGLiberal says:

    @Kay: Agreed…the fact that you have to register just to see the options available to you is crazy. That was the first thing that struck me as ridiculous. Forget the fact that it took days for me to actually get through the registration (for over a day the “secret question” drop-downs were blank so couldn’t complete that step) and that when I did it didn’t seem to accept my login, saying it could find my profile, but when I tried to re-register it said I was already registered, etc. – why did I have to do all of that to see what options were available to me.

    My guess is that they wanted to make sure they could collect our names and email addresses so they could send the FEMA storm troopers and black helicopters to our houses and send us to Obama Kenyan Socialist Muslim re-education camps.

  31. 31
    Corner Stone says:

    God dammit Chuck Todd! Tell Bill Fucking Kristol to shut the fuck up and stop talking over every one else there!
    You lying sack of shit!

  32. 32
    Tommy says:

    @TAPX486: Now I totally understand hindsight is 20/20, but the first thing they ask a user is their state. I am thinking I might have hired a team to manage the site by state.

  33. 33
    Keith P. says:

    For those who have ideas of programmers being some elite cabal of super-smart nerds, let me dispel that. I’ve interviewed dozens of them over the years (am one myself), and *easily* more than half are completely unqualified for the positions they are interviewing for. As in “I’ve got X years experience designing ASP.Net web apps” “OK, what would you use to login to one of these” “Hmmm, I haven’t had to write that part before.” Or “Generated a random password (just a string of random characters” after which, the candidate sits there with a marker in his/her hand for a minute or two before giving up.
    I am surprised not one bit that the rollout went south like this (especially given that government contracts pay less, so you’re getting the candidates who couldn’t get hired on to better-paying contracts in the midst of a good job market – software development is booming right now)

  34. 34
    Belafon says:

    As a software developer who has worked at both a government contractor companies and companies that do not do contract work, I’ll tell you that there really isn’t much difference in the quality of software. Wasn’t it the Sims online that also blew their rollout by not properly anticipating load?

    The company I work for is certified CMMI Level 5. They apply it to everything from their manufacturing to their software development. I’ve been here for two years, and I’ll tell you that the software quality hasn’t been helped by the process.

    On the other hand, my last company had the same issue, and they weren’t following the same process.

  35. 35
    jw says:

    I work as a web developer for a consulting firm with several government contracts. It is easy to be snarky about lazy government employees and greedy contractors, but the truth is actually fairly mundane and boring. In my experience, the government project officers charged with the development of these sites are policy experts with little or no experience in web site development. Consequently the project suffers from lack of direction, poor allocation of resources, and mission creep. The same problems occur in the commercial world, but it is exaggerated in government projects because the project officers do not have the necessary experience to lead these projects.

  36. 36
    Robert Sneddon says:

    @WereBear: This. In a few days or weeks the traffic to the assorted healthcare sites will die down to a steady lower level with maybe some peaks here and there through the year. Building out the sites to deal with the gigantic surge in interest for the few days after the exchanges opened would be like building a 6-lane highway for a route that only has a few thousand vehicles per day pass along it except for the first few days after it was opened.

    Saying that it might have been better to build two sites (and possibly some states did so), one site handling general enquiries and folks interested in price comparisons and another for people ready to put their money down. The annoyance would be from people expected to enter lots of personal and financial data twice if they did decide to go ahead with an exchange policy.

  37. 37
    Corner Stone says:

    Jesus. First Bill Kristol and next Chuckles is having Mitch Daniels on to speak about education costs!
    What freakin channel am I on again?

  38. 38
    TAPX486 says:

    @Botsplainer: If I remember the F111 was also supposed to be a multi-purpose/multi-service/multi-nation one plane fits all sizes. I think it started life as the TFX fighter. The Swiss army knife of airplane design certainly sounds like a good idea but I suspect once you get something more complicated than a Swiss army knife the concept breaks down

  39. 39
    The Red Pen says:

    I’m doing government contract programming (at least until Friday, when the fund that pays me depletes*).

    1. Depending on where you are, there may not be any kind of functional software development culture. For example, Air Mobility Command is a huge operation and has the software development culture of a 90s-era small insurance company. Under these conditions, really shitty software development can occur without consequences. Large sections of the code base I started working on earlier this year could have been written in 2002, but they were written in 2012.

    2. Staffing can be difficult. Everyone on my projects needs a security clearance. Part of the reason why our Java code looks oddly retro, is that most of the developers learns their Java ten years ago and haven’t updated their skills. Experience is great, when it’s combined with a desire to stay current. We had some young guns who follow the new stuff and understand agile methodologies really well, but it’s hard to get that many. When I was their age, I couldn’t have gotten a security clearance and, in fact, didn’t want to try for one.

    * This fund also supports military humanitarian aid shipments, so I’m praying that if it runs out, Steve King’s (R-IA) district will be devastated by an F5 tornado.

  40. 40
    Elizabelle says:

    @Corner Stone:

    Not Chuck’s job.

    If Bill Kristol is messaging the best, Kristol deserves to shout everyone else down.

    Chuck told us that.

    PS: I am staying off MSNBC too this week, although I do like what I see of Martin Bashir.

    Why can’t he have an evening show, instead of rerunning the other evening shows? Bashir is more news roundup than opinion, opinion, opinion and — honestly — Rachel’s jeering.

    Chris Hayes has literally been shrill of late. Talking voice quality only.

  41. 41
    Belafon says:

    @Keith P.: I’ve had the same issue. “Please write a function that returns the nth Fibonacci number.”

    “Generated a random password (just a string of random characters)”

    ***************************

  42. 42
    Kay says:

    @OGLiberal:

    Also. Making the call center people dependent on the website? I mean, for goodness sakes. Can they not put the insurance information on their machines off-line? They’re the call center. The people who call aren’t using the website.

    They should have put social workers and second grade teachers in charge of this. Do they deal with the actual public, the people who built this, instead of some idealized version of the public?

    Have they ever been to a local SS office? Hang around there ten minutes, Listen to the questions they get asked. They have to be extremely patient people.

  43. 43
    BigHank53 says:

    Bought the stuff from the lowest bidder, didn’t they?

  44. 44
    Mark B. says:

    Load testing software is a pretty well-known technology, and it’s something we did regularly. You have automated tools that can generate a very high number of web requests from a relatively small farm of test machines. It’s really a mandatory part of building a high-demand website, or any other kind of application which uses servers. I worked as an (in-house) programmer for a state government, and we were pretty committed to doing this. It’s amazing that this kind of work wasn’t done for this site, but it’s probably due to outsourcing the job to contractors instead of using dedicated programmers who stand behind their work. Contracting government work to outside contractors always leads to lower quality work. Always.

  45. 45
    gene108 says:

    whole government contractor grift,

    Government contracting is no more or less a grift than any other kind of contracting.

    You have good general contractors, who would finish your home renovation within budget and the time allotted and others, who’d try and fleece you for as much as possible and do shoddy work.

    Same is true for people working on government projects.

    There’s nothing inherently bad about using contractors. There’s just some shit that does not make any sense to have full time people for, because you do not have full time work for them to do. Once the project is over, you don’t need them anymore.

    Anyway, until there’s some actual investigation on why the healthcare.gov sucks balls for enrolling, I’m going to offer my own bit of speculation: They didn’t fucking realize most of the states would refuse to set up exchanges and are handling a larger load than expected, not just in volume but in every sense imaginable.

    The ACA gets passed in 2010.

    States are supposed to set up their own exchanges.

    A bunch of states sit around not making any statements about setting up or opting out of exchanges, until the SCOTUS decision on the case, which I believe was handed down just last year.

    They then make their decisions.

    IT projects suffer from scope creep all the time. Having stakeholders in the process introduce scope creep, a little over a year before the site goes live doesn’t help.

    Plus GOP refusal to give HHS the funds needed to implement the law doesn’t help. From a May 2013 WaPo article:

    Health and Human Services Secretary Kathleen Sebelius has gone, hat in hand, to health industry officials, asking them to make large financial donations to help with the effort to implement President Obama’s landmark health-care law, two people familiar with the outreach said.

    Her unusual fundraising push comes after Congress repeatedly rejected the Obama administration’s requests for additional funds to set up the Affordable Care Act, leaving HHS to implement the president’s signature legislative accomplishment on what officials have described as a shoestring budget.

    Link

    So you have a bunch of (a) states deliberately waiting till the 11th hour to decide to set up exchanges or push it off on the Federal government, (b) more states than expected, when funding for the law was originally established pushing work of setting up exchanges onto the Federal government and (c) the GOP refusing to allocate more money to address these issues.

    In terms of IT projects, what could possibly go wrong with an underfunded project that has a lot of last minute scope creep and is handling more than expected, when it the first plans were drawn up?

    Honestly the website problems are just another symptom of the GOP sabotage strategy.

  46. 46
    Tommy says:

    @The Red Pen: I think your points are pretty darn valid. I do web sites for a living but I don’t consider HTML/CSS/PHP writing code. In another time in my life I was the VP of marketing for some software companies and often I was asked to work with the tech department, cause well I could work with them.

    It is my experience that folks don’t understand how software works, and often those they hire to write said software are not so good at their jobs.I actually left my last job cause well our product didn’t work that well. I felt the tech people had the owner handcuffed cause they didn’t understand what they were doing. And they were scared to death to let them go and not know who would write the code.

  47. 47
    TAPX486 says:

    @Tommy: True. But I guess having lived thru Windows 3.0 all the way to Windows 8 and how many versions of OFFICE before they standardized all of the key combination I’m willing to give them a little bit (and the operative word is little) slack. If the same issues are still happening in Nov. then that is a different story. I also understand the visuals on the Presidents signature achievement

    As for the volume yes they should have designed for a higher level of traffic given that 30 million people are eligible to sign up. Maybe they didn’t realize that all 30 million would try to do it at the same time. :-)

  48. 48
    cvstoner says:

    Well, I DO deal with government contracting, and the answer is simple. If you want the government to produce better IT solutions without having to use contractors, then pay government IT people a competitive salary to stay and build them. It is as simple as that.

    Starving the beast has hit government IT departments as hard, or harder, than anything else, causing a mass exodus of talented people, thus forcing the government to seek help from the IT contractors where many of these talented people have landed.

  49. 49
    Kay says:

    @OGLiberal:

    That’s what I mean abut the arbitrary nature. For some reason, the security questions work fine for me. Not for anyone else. Just me. My problem is the verifying email, but I may have beat it!

    That’s why they should stop saying it’s volume. They should stop saying it’s volume anyway, because no one cares why outside Demos. They should say “it doesn’t work and we’re fixing it”.

  50. 50
    Steeplejack says:

    @OGLiberal:

    I have tried multiple times over the last week to sign up at HealthCare.gov, including several times between midnight and 3:00 a.m. EDT, and I have never made it past the first screen of “Stay on line, we’re really busy” or occasionally “System is down for scheduled maintenance.” They should mix it up with this screen for entertainment value.

    I am a big fan of the ACA and think this will really work, but the Web site problems are starting to look like more than just “swamped by opening-day traffic.”

  51. 51

    For some reason I got on the mailing list of a right-wing outfit called the Laissez-Faire Institute and it’s absolutely hilarious this thing they sent me about Obamacare:

    It’s already being reported that exchange websites have crashed, and that the premiums are actually more expensive.

    Everyone knows those things now.

    But what no one’s reporting is that the Affordable Care Act is a merely a front for something much, much worse.

    Ooooh what could that be, I wonder?

    You likely won’t see this story reported on any mainstream media sources – it’s too controversial even for the most conservative talk shows.

    But I assure you, months of research point a massive crisis in the making.

    Massive!! They’ve gotta scoop! So hot they gotta wear gloves!

    What’s coming could be much worse for your family’s health – and wealth — for decades to come.

    We’ve put together a website that shows you what’s coming… when it’s coming… and how to “opt-out” of this next crisis.

    Despite us investing more than 1,000 hours of time, and costing us nearly $100,000 in research fees to create this website… you can view it for free, right here.

    Wow, that’s real swell of them. A website you can visit for free.

    I actually haven’t visited the website because I don’t want to give them the hits. If anyone is brave enough to do so, please by all means help yourself. I’ll send you this email if you give me an addy.

  52. 52
    sdhays says:

    I used to work on a software project at one of the non-DoD departments. There was no-one on the government staff who knew anything technical about the system; for anything technical, they were completely dependent on the contractor team. For some perspective, my former team is considered “essential”, so this wasn’t some small pet project. Most of the government managers were either unengaged or hyper-focused on making sure we followed some process from DoD that they didn’t understand and didn’t make sense for our project and trying to make sure that every flaw resulting from that process was blamed on the contractor. Even though it was their software, they took no responsibility for it; their attitude was that we were developing shrink-wrapped software rather than simply building the software they told us to build.

    Basically, even if you have contractors giving good advice and caring about developing a better product, the way the government is setup is to avoid having competent, knowledgable people overseeing the process. On top of that, having all your knowledgeable people as contractors means that they are purposefully left out of strategic decision-making; you can guess how that works out.

    On my project, the contract was changed and rebid and my company lost. My company was a small company which had built the software from its inception (before I joined). They bid too high, partially because they actually knew what it would cost to do what the government wanted to do and they wanted to do a good job. The company that won was a complete disaster (I know because I switched employers to stay on the contract). While the government saved some money on paper, I know my former employers’ plans and preparations would have made a much, much happier customer.

    I’m happy to be back in the private sector. The dysfunction by design was really draining. And, yes, the private sector can be stupid too, but the structure of government hiring and contracting is really setup to fail.

  53. 53
    sdhays says:

    @cvstoner: Yes. And on top of that, the emphasis seems to be eliminate those types of positions altogether. The perception is that all we need are managers and let the contractors do the grunt work.

  54. 54
    Tommy says:

    @Mark B.: “Load testing software is a pretty well-known technology [….]

    That is an understatement.

    Now I am a hippie liberal and I spent hours trying to get onto the site. I ran into all the problems mention here. But I work for myself and wanted to know what options I had over the terrible plan I now buy. It took me hours and hours. I tried to log-on like a few dozens times, even at 3 AM when I hoped nobody would be there.

    I will save almost $100/month. I am pretty happy. I just fear folks not like me. Maybe not liberals, will visit and get mad and never come back.

  55. 55
    Elie says:

    There are many exagerrations, inaccuracies and misstatements in this post. Many government agencies have been stripped down of their experienced employees over the last 10 -20 years. Did anyone of you so called lefties notice? Everyone hates government – the left and the right. Right now, they have been under hiring freezes for a year. The load of many contract specialists is huge, but most try to do a good job in selecting contractors but monitoring them is difficult given their loads. Private software contractors are not all crooks or incompetents, but there is frequently a fair amount of uncertainty and bugs to be worked out with any implementation.. you can see this in the private sector as well.

  56. 56
    Bill E Pilgrim says:

    @Southern Beale:

    Laissex-Faire Institute

    Hmm. I’m pretty sure the one who mailed you is the Laissez-Faire Institute.

    Laissex-Faire is a Rennaisance Fair for people into latex.

    Well, that’s what it sounds like it would be, anyway.

    Edit: D’oh! No fair still having edit time on yours before I made my joke. Shakes fist.

  57. 57
    MattF says:

    @Southern Beale: Hmm.

    it’s too controversial even for the most conservative talk shows.

    So, like, more controversial than the coming of the Antichrist and the end times? I’m really sort-of curious, but no, I’m not going to try to find out.

  58. 58
    cvstoner says:

    @Another Botsplainer:

    If the taxpayers are paying a GS13 programmer about 100K a year, they are paying 3x to 4x for a contractor to do the exact same work. But it’s all good because the contractor is not a lazy government freeloader.

    Which government are you talking about? Sweden? In most state governments, the IT supervisors aren’t making that much! Most of the grunt developers are making $50-$60K.

    Also, the state governments are forced to use contractors because the IT departments are basically running skeleton crews. The work has to get done somehow.

  59. 59
    Tommy says:

    @cvstoner: Very true. A good software developer is worth their weight in gold. I am not sure the government has a GS payscale high enough to hire the best of the best. In fact I know there isn’t.

  60. 60
    cvstoner says:

    @Elie: Yes. Exactly.

  61. 61
    PurpleGirl says:

    @TAPX486:

    The contractor would promise, for a dirt cheap price, …

    I know that in NYC, when contracts are being competitively bid, it is policy (and maybe law) that the City has to take the lowest bidder. They are assuming that all other factors are equal, which, of course, they rarely are. And then when the price goes over the estimate, the City still pays the overage. So much for the lowest bid.

  62. 62
    Keith P. says:

    @Belafon: My current company uses the Fibonacci problem for its “advanced” interview question if they get our “count the number of digits to the left of a decimal” problem. Shocking how few people get the latter, and we’ve had maybe one or two get hte Fibonacci. The problem I have with those questions are that we interview for ASP.Net jobs, and we end up getting people who may be good programmers, but they know squat about the .Net libraries, so they’ll spend weeks writing code that does stuff built into the framework. Drives me absolutely bonkers to see time wasted like that .

  63. 63
    Steeplejack says:

    @Kay:

    But you have to register to get in at all.

    They could have taken care of 90% of the people by just letting them look at what’s available.

    This. Get a few essential facts—age, sex, whatever—and let people look at samples of the plans. And it would be okay to put up notices that “Actual plans/rates may differ when you actually register and enter all your information.” But people would understand that; they just want to check things out at first.

    By (apparently) front-loading the whole registration process they have really mucked things up.

    And ditto on your point that a lot of “customers” are not going to be super sophisticated users of either insurance or the Internet.

  64. 64
    Elizabelle says:

    WaPost: As of 9:30 this morning:

    The 14 House Republicans supporting a clean CR are:

    Scott Rigell (R-Va.)
    Pat Meehan (R-Pa.)
    Charlie Dent (R-Pa.)
    Peter King (R-N.Y.)
    Devin Nunes (R-Calif.)
    Jon Runyan (R-N.J.)
    Mike Fitzpatrick (R-Pa.)
    Frank Wolf (R-Va.)
    Michael Grimm (R-N.Y.)
    Rob Wittman (R-Va.)
    Erik Paulsen (R-Minn.)
    Lou Barletta (Pa.)
    Randy Forbes (Va.)
    Frank LoBiondo (R-N.J.)

    Five of them — including Dent and Meehan — come from the Philadelphia area.

    Call a few, and tell them you appreciate their support.

    I just got off the phone with one congressman’s office; young man did not ask if I was a constituent, but was happy to hear the thanks.

    He said they were getting calls of thanks, more calls of people saying this is “too little too late,” and complaints from a “vocal minority.”

  65. 65
    amk says:

    @Elie: Yup. Plenty of belly aching with stereotyping.

  66. 66
    cvstoner says:

    @sdhays: Exactly. The breadth and depth needed to build these applications without the help of contractors is no longer there. Most government IT departments are there for maintenance and tech support.

  67. 67
    dpm (dread pirate mistermix) says:

    @Elie: I don’t think the private sector is immune from exactly the same phenomenon of blame-shifting. In fact, I think that large private sector corporations have bureaucracies that act the same way.

  68. 68
    amk says:

    @Elizabelle: Excellent post.

  69. 69

    By the way, did you know that the House gym is deemed “essential”? Me neither.

  70. 70
    aimai says:

    @Kay: Yup. I don’t fault the government exactly–they had some pretty fucking strong headwinds against being allowed to set this up at all. But it is basically the case that they are offering an entirely new service to people who are not used to getting any service at all or a service which, traditionally, has been handled by a middle man (the insurance agent, the company) so that the end user never even ends up comparing plans.

    Why they didn’t go to Mass for a model I don’t know. The MA exchanges have been up for six years. The idea that they had to do this all for the first time is beyond me.

  71. 71
    hoodie says:

    @gene108: I tend to think this is the issue, unless proven otherwise. My admittedly dated experience in government contracts is that stuff that gets initially lowballed sucks when first released. The lowballing can come from a government agency that is budget constrained. The GOP deliberately sabotaged the original architecture, which was to rely on the states, and the obviously has exacerbated the problems.

    I thought the news was that several of the state exchanges were operating pretty well. Those may be the saving grace because, if there are enough successes, it could provide evidence of how well the system can work if the GOP would quit ratfucking it. If I were Sibelius, I would emphasize those state exchanges, because it would dovetail with the GOP-initiated shutdown and other deliberate attempts to sabotage the operation of the country.

  72. 72
    taylormattd says:

    I’m finding this obsession of yours to be a bit baffling, especially given it flat out reinforces the wingnut narrative of government failure.

    I also find it baffling because of the sheer number generalizations, assumptions, and the conclusion-jumping.

    There are what, 50 different websites? Did they all use only private contractors to build their websites? Is the same private contractors? Are they all terrible because they are private? Are states never supposed to use private web developers? Are they supposed to retain a department of web developers? (Idk, maybe the answer to that is yes)

  73. 73

    @Elizabelle:

    Only three from the South, the Virginia delegation. I hear Virginia has been really hard hit.

    Well that figures. The Confederacy drags America down yet again.

  74. 74
    cvstoner says:

    @Tommy:

    A good software developer is worth their weight in gold. I am not sure the government has a GS payscale high enough to hire the best of the best. In fact I know there isn’t.

    Nor do they need to. The money would be better spent on increasing the depth and breadth of the IT department, and then bring in the “best of the best” for the short amount of time they are actually needed.

  75. 75

    @Bill E Pilgrim:

    Typo. I fixed it.

    X is next to the Z on the keyboard. Give me a break.

  76. 76
    Tommy says:

    @PurpleGirl: I mentioned at the very top here my first job out of college was working for a market research firm that tracked federal procurements. I speak procurement speak. I can’t speak to states but the Federal government is mandated by law NOT to buy the lowest cost bid. I mean that in a good way. They have to factor in what is called past performance and the quality of the RFP. Also 8A awards for minority or disadvantage firms.

    What you said was the norm. Folks would underbid a contract to win it then not produce. That is why the past performance thing was thrown in. That you couldn’t win a contract where you underperformed on other contracts.

  77. 77
    Alex says:

    There are a lot of issues with the site, but they will probably be fixed within a month. There’s no difference between signing up now and signing up on December 1st. Which is what they should be saying on every single error message.

    From what I’ve read, it’s not loads that is the issue — it’s a lot of different sites and databases all needing to be checked. And the site is not set up to put a “We’re checking your immigration status” flag on the application.

  78. 78

    @MattF:

    I think it’s like that 11th level of Scientology where you have to spend a lot of money go through years of special brainwashing before you’re even capable of receiving information this explosive.

    At least, if they were smart that’s what they’d do. They’re just trolling for donations, anyway.

  79. 79
    Belafon says:

    @Keith P.: Totally understandable what you need. It kind of depends on the context. When I was interviewing for senior C# or MFC developers I would ask library questions.

    I don’t know where you work, but I’m in Dallas. I did an interview once in Seattle to work for Amazon. They didn’t offer me enough to move, but I attempted to use some of the same questions I was asked here in Dallas and had to stop: The questions pretty much floored most of even the senior developers I interviewed. Most developers just don’t keep improving their skills, and most software companies don’t really encourage it.

  80. 80
    PurpleGirl says:

    @Kay: Yes, you’re right. I’m used to reading web sites before a make a large purchase. I read and compare products and then buy, either on line or in a store. I don’t drive so I don’t need auto insurance. But I’ve needed apartment/house insurance several times in the past. I’ve done research on policies and companies. My current policy I bought over the phone and renew on-line. It’s easier if I can research first and place the order later.

  81. 81
    Kay says:

    @Steeplejack:

    And ditto on your point that a lot of “customers” are not going to be super sophisticated users of either insurance or the Internet.

    People are generous. Not all of them, we’ll say 27% will be assholes who want things to fail, but MOST people want a big ambitious risk to succeed, particularly if the objective was to help them.
    But you have to tell them it’s broken and then fix it. No one who matters in this will crucify them for that. But this sort of snippy, defensive “it works fine without load” is just a bad approach. I know the guy made it and he’s invested in defending it, but he has to drop that. It isn’t about him. If they’re telling him it isn’t working, then it isn’t working. It doesn’t have to work for HIM. He has health insurance. It has to work for a random 31 year old low wage worker who doesn’t have internet access and has never purchased an insurance product before, outside of state minimum car insurance. That’s (eventually) their client.

  82. 82
    Tommy says:

    @cvstoner: I don’t like the outsourcing. A good friend of mine works for Unisys. Guess where he goes to work each day, the ATF headquarters. Runs their help desk.

    Now I like my friend. I also like Unisys. Was a client of mine and I got him that job interview, but I don’t know why they can’t run their own help desk.

  83. 83
    boatboy_srq says:

    @Another Botsplainer: There’s a lot of overhead in IT consulting, whether contracted to the public sector or to the private. The expected economy stems primarily from the perceived temporary nature of the contract: the client is not expecting to cover the intangibles of employment (healthcare, retirement, liability, expenses) and can afford to pay more in the short term anticipating those savings in the long term. The one key difference I have seen between the public and private sectors is that the public one has greater oversight and higher accountability standards: probably 1/3 of the cost of a public sector contract goes to satisfying all the regulatory and compliance requirements in the contract, which the private sector can either handle internally or forego entirely.

    The trouble with US federal public-sector contracting is that it increasingly doesn’t work quite that way. The employees come and go, but the contract persists. One look at Lockheed, Raytheon or SAIC is enough to illustrate that. And while private industry is valuable in the delivery of physical and intellectual product (imagine what the space shuttle would have cost had NASA and USAF built it all themselves) the lifespan of an IT deliverable makes measuring the costs harder since it depreciates faster: the public sector is still unaccustomed to the kind of lifespan that a Website experiences, or that the underlying hardware and software required to deliver it.

    Add to that the premium required to pay skilled labor in the areas where public-sector contractors congregate and the pricing jumps significantly. Metro DC pays very well, as do Silicon Valley, metro Boston and other areas where the businesses that cater to the federal government tend to congregate. The pay scale – especially for this project – is likely skewed by the number and types of professionals involved. The skills required for most of the work on a public-facing Web project in metro DC fetch low six figures at least, and there are HIPAA and assorted other security requirements that will add to that number. Those dollares aren’t especially poorly spent.

    For old-school conservatives, the contracting environment is feature-not-bug territory: the private sector is by definition more efficient – and therefore more economical, and therefore more virtuous – than the public sector, and more worthy of the expenditures than its public-sector equivalent. I can’t help wondering how the Teahad views the equation, though: it’s still taxpayer money being thrown at a particular project/sector, all of which fuels the deficit which they all claim to hate.

  84. 84
    Eric U. says:

    government software jobs tend to be very large and almost never work out of the gate. There are some really well-known examples of this in the past. All done by contractors. I really don’t think this is the sort of job that the government would do well anyway. I worked at a place that was trying to do a large software job, it actually worked, but I don’t know if it’s cheaper than having contractors do the work. The place was stuffed to the gills with contractors anyway.

    What I hate to see contractors doing jobs that really require some sort of dedication to the job. Like logistics. I knew someone that took a job with a contractor, and his job was to make presentations for government workers. That was nuts, but those kinds of people are all over the place now.

  85. 85
    Kay says:

    @PurpleGirl:

    Insurance is a complicated product. It doesn’t have to be excruciating for them, either. Talking to the 21 year old night before last was interesting. He has to think about things he has never considered before; how often does he go to the doctor, how much does he think is reasonable to pay for visits, etc.

  86. 86
    Bill E Pilgrim says:

    @Southern Beale: Just having fun, of course it was a typo.

  87. 87
    magurakurin says:

    you know what. It’s gonna get fixed. Nobody can actually get a policy until January. By December these problems won’t even be a memory. It’s gonna get fixed. These problems are much better to have than nobody trying to sign up. It’s going to work. By this time next year, ACA will be an unqualified success.

    Bookmark it, libs.

  88. 88
    Chris says:

    @becca:

    Interesting thing I remember reading about the Washington Consensus; it was sold in third world, heavily statist economies as “give your inefficient public-run corporations to the private sector, then they’ll suddenly become more efficient.” In reality, the private sector was only interested in public-run corporations under one of two scenarios; 1) if they were going to break it up and sell it off one piece at a time, or 2) if the corporation was already turning a profit. In other words, only in cases where the “they’ll turn things around!” premise didn’t apply.

    Economics has become theology, as much as it was in the old Soviet bloc. Maybe it’s time to bring back the “it doesn’t matter if the cat is black or white as long as it catches mice” slogan.

  89. 89
    RaflW says:

    Ohhh, so that’s where Mitt Romney’s campaign IT department landed!

  90. 90
    RobertB says:

    I had an…okay?….experience? Last Saturday, around 10 AM, I signed up, just to see the prices. It took about 20 minutes, but after the 20 minutes I was in. No glitches, no nothing but a bunch of questions like, “What size brogan does your grandmother wear?” However, this is all at the intersection of Government, PHI, and Insurers, so I’m not surprised that there’s a lot of red tape.

    The prices themselves didn’t seem horrible, but weird. Some companies’ Bronze plans cost more than others’ Silver and Gold plans. I suspect that’s due to the size of the provider network; I know a little bit about the health insurance industry, but not about how provider network size impacts premiums.

  91. 91
    joes527 says:

    @Alex:

    There are a lot of issues with the site, but they will probably be fixed within a month. There’s no difference between signing up now and signing up on December 1st. Which is what they should be saying on every single error message.

    This.

    If this were a private enterprise, the cost/benefit analysis of dealing smoothly with the initial surge would have come back: “not worth the cost,” and they would have had exactly the same problems.

    But because this is the government, this is an unprecedented disaster.

    meh.

  92. 92
    Steeplejack says:

    I had (another) “Outsourcing—bah!” moment yesterday when I paid my Fairfax County (VA) auto tax on line. The Web site is fairly convenient and easy to navigate, but if you want to pay with a debit or credit card they tack on a hefty fee ($6 or $7) that goes to a private company for “processing.” How does this make sense vs. (a) doing the development in house or, if that’s not possible, (b) paying a private company to do it as a stand-alone project with no endless revenue stream on the back end?

    I, like most people, probably, am happy to pay a few extra dollars for the convenience of paying on line, but I would much rather see that money go to the county itself. But in this privatized setup some company is getting a huge ongoing royalty for work long since done (except perhaps for some minimal maintenance).

    Having said all that, there was an option to have the money yoinked directly out of your checking account, and for that there was no added processing fee. WTF? (That’s the option I took.)

    And, on a semi-related note, and related to our recent discussion about the ridiculous ID hoops you have to jump through to get a driver’s license, etc.: If you need to get a bone fide copy of your birth certificate and try to do it on line, in many states you are funneled through to a quasi-official Web site where, again, a private company has its probe firmly inserted into the money stream. The price for a birth certificate seems a little high just in itself, but their particular “innovation” is that the “regular” mail option sounds vague and sketchy, as in you don’t know when you will get your birth certificate, but you can pay extra to get two-day mail—and you know a lot of people are needing a birth certificate because of some sudden mini-crisis, not because one day they were sitting around and decided they really needed to update their personal dossier. And the two-day mail thing was a lie, in my experience, or else they took a really long time to hand-tint my birth certificate and ink in all the fancy curlicues before they put it in the mail.

  93. 93
    Mayken says:

    @joes527: This! A thousand times, this! If I had a nickel for every project that executives underfunded and then had to throw more money at to fix after the “go live” I’d be retired by now.

  94. 94
    Mobile Grumpy Code Monkey says:

    Speaking as a software engineer with over 20 years’ experience, overestimating capability and underestimating demand is simply a fact of life in any large deployment. Contractor culture, outsourcing, et al. may have played a role, but issues like this are endemic to large systems, private or public. I don’t think many people realize just how complex a system like this can be.

  95. 95
    joes527 says:

    Anyone who wants a point of comparison with the private sector, go try to get technical support from your cell phone company.

    I’ll wait.

  96. 96
    Keith P. says:

    @Belafon: I had an interview before that was supposed to go 90 minutes but went 3 hours because the interviewer started getting into deep .Net (like what certain IL looks like), and I was keeping up with him. I really enjoyed it because I got to use some really obscure bits of .Net knowledge that almost never comes up (I try to know it inside and out and am always going after my weaknesses). Then I started the job, and I started seeing bizarre code like event handlers that contained lambdas that just called other methods for no apparent reason (i.e. private void HandleMouseClick(…) { (x) => (SetEnables()) }). They’d hired some devs who didn’t understand the featureset and started using constructs just for the sake of having them in there. I’m not very diplomatic about stuff like that…stupid is stupid.

  97. 97
    Elie says:

    @dpm (dread pirate mistermix):

    Yep — I know cause I am in one right now (private company bureacracy).. absolutely the same way. Ummmm — could that mean that human beings tend to behave similarly in large organizations implementing something big and complex? Could it be?

  98. 98
    Elie says:

    @magurakurin:

    Agree with you…

  99. 99
    Mark B. says:

    Are states never supposed to use private web developers? Are they supposed to retain a department of web developers? (Idk, maybe the answer to that is yes)

    I think the answer to the first question is ‘not always’. If it’s a site that’s just a one-off and isn’t part of a regular business process, why not just hire a contractor or get a pre-built product. But most of the sites that governments product are central to their business process ane are integrated into their internal system, either as providers of data, or as readers or data (or maybe both). And, in my experience, coding a web site is really just a beginning. Most web sites need constant maintenance as requirements change and you find better ways to do things. If you buy something from a contractor, you are going to end up spending a fortune paying them to maintain it. In almost every case, if the agency is large enough, it makes sense for them to have their own dedicated staff to program and maintain their own web sites. In some small states, you might need a combined IT service that covers a lot of customers (agencies).

    At the federal level, it shouldn’t even be a question of whether they should have their own internal staff to handle IT. It’s more efficient, and cheaper in the long run. They can hire contract employees to handle load at high demand times (rolling out a new product) but they need in-house expertise.

  100. 100
    Steeplejack says:

    @TAPX486:

    And a Swiss army knife isn’t usually that great, except as a convenient something you might have on hand when you don’t have the “real” tool.

  101. 101
    Mark B. says:

    @Keith P.: I’ve got mixed feelings about lambda expressions. They are super cool. I actually learned programming using MIT Scheme and have been using them since the 90s, I was elated when they finally became part of the .NET framework. But they’re so easy to misuse to create unreadable code. It’s not always necessary to create the tersest possible code. Sometimes more verbose code compiles just the same and is much more readable.

  102. 102
    Elie says:

    @Eric U.:

    Don’t know where you have been or your experience, but I would say that MANY large software jobs in the private sector (data warehouses, big databases, web sites, workfflow systems, etc) also frequently have implementation bugs that need to be worked out. Of course, you don’t have media looking over their shoulders for every glitch, so these tend to be under reported unless you know by having direct contact with the project, as I have. Trust me — from hospitals, to businesses to academic organizations — ALL have had problems during big implementations. Its just that the government gets the scrutiny that the others do not and they have the kool kids like you snipping about how the private does it better cause you know its private. NOT

  103. 103
    magurakurin says:

    @Steeplejack: and the most useful tool, the tweezers, always get lost within a month after purchase.

  104. 104

    @Bill E Pilgrim:

    Although a Laissex Faire does sound like a helluva lot more fun ….

  105. 105
    hoodie says:

    @Mayken: Yeah, that’s cultural phenomenon widely found in American business, give your people a chair with one leg shorter than the other and that will somehow make them more resourceful. It’s driven by management’s need to push favorable numbers to make a sale but often it’s just received wisdom and macho bullshit. It is not always the case, however, as some managers are more creative than that. I doubt it’s the case here except, perhaps, within the contractors’ management. I would imagine the funding for this was strained and they ended up having to do more than they planned with limited resources because of the failure of several states to cooperate. They just needed to be online on Oct. 1 and let at least some people get signed up and hope the blue state exchanges can do better.

  106. 106
    Steeplejack says:

    @gene108:

    You make really good points.

  107. 107
    Belafon says:

    @Mark B.: I think of lambdas in OO languages as “finally, functions are first class objects.” As for what you described, I’m pretty sure you saw that problem with anything. Most people don’t really have the discipline to write a lot of code, and other than providing better examples, I don’t really know how to change it.

    As for Keith P’s example. Gaaah. It’s obvious they don’t quite know how event handlers work.

  108. 108
    Mike in NC says:

    I recall an old joke from the Beltway Bandits that I worked for: “Better, Faster, Cheaper. Pick 2”.

  109. 109
    The Red Pen says:

    @ Keith P.:
    @Belafon:
    My rule is never hire anyone who doesn’t know what a regular expression is, or can’t handle some basics in the area.

    Keith’s point about libraries is really key — especially in this day where there’s an OSS project for pretty much everything. At this point, I consider the Spring Framework a core API for any “enterprise” Java program. I know there’s a Spring.NET, but I don’t know if it’s as advanced. Anyway, people who don’t know Spring at pretty much doomed to rewrite it — usually badly.

    Bottom line is that there is so much demand for software developers, that there are a lot of people who can draw a paycheck with the really poor skills and expertise.

    @cvstoner:

    In most state governments, the IT supervisors aren’t making that much! Most of the grunt developers are making $50-$60K.

    In the federal sphere, most developers are GS-12 or GS-13. Any lower than that and there is too much turnover.

  110. 110
    pseudonymous in nc says:

    @Mark B.:

    Load testing software is a pretty well-known technology, and it’s something we did regularly. You have automated tools that can generate a very high number of web requests from a relatively small farm of test machines.

    Oh, c’mon. Real-world load testing is hard. It’s not just a case of blitzing your servers with basic requests for anonymous traffic. You have to simulate user activity — which in this case means handling submitted form requests — then scale it up. And even the best simulations aren’t great at exposing the weird errors that only occur on high load.

    This doesn’t mean that it can’t be done, especially if you have money in the bank, but it’s not just a case of running siege and seeing if you hit max clients or are CPU-constrained.

  111. 111
    Mark B. says:

    Oh, c’mon. Real-world load testing is hard. It’s not just a case of blitzing your servers with basic requests for anonymous traffic. You have to simulate user activity — which in this case means handling submitted form requests — then scale it up. And even the best simulations aren’t great at exposing the weird errors that only occur on high load.

    Yeah, you can’t just do an ‘out of the box’ test with a standard product, but really building a script that does things like transactions and verifies the results isn’t all that hard. Yeah, you won’t handle all of the edge cases, but you should be able to generate a realistic amount of load using the normal business process without too much effort. Even if you’re not using one of the common test suites, creating web requests and listening to the responses is pretty basic stuff. Yeah, you’re still going to sometimes have weird problems in the real world which your testing didn’t anticipate, but you need to do your due diligence and test as much as you can. I don’t see much evidence this was done here.

  112. 112
    👾 Martin says:

    The two people I know who are working on this tell me something a bit different. Much of the problem is on the legacy systems that the exchange software needs to interact with to determine if you’re Medicaid eligible, etc. plus interfacing with all of the insurers different systems. In both cases, systems are out of date and not geared for this sort of thing. The government systems are the usual clusterfuck of never being working because Congress changes the rules faster than anyone can implement them, and you’re relying on private contractors like HP to implement them.

    Load testing isolated systems is pretty easy. Systems like this which delegate a lot of the business logic to legacy systems and systems out of the control of the designers (all of the checks with the hundreds of insurers themselves, the states, and so on) are nearly impossible to load test because it requires building an entirely redundant system that can create accounts, build policies, check incomes (on real people, mind you), and do all of that with no adverse consequences (such as actually signing someone up for a policy under testing situation).

  113. 113
    Mark B. says:

    @Belafon:

    As for what you described, I’m pretty sure you saw that problem with anything.

    You’re not kidding. LINQ expressions are one of the worst examples. You can sometimes replace 10 lines of code with a single LINQ expression, but it’s not always a good idea. Sometimes it is, if the expression is logical. Another pet peeve is a predicate with dozens of clauses. Sometimes it’s much more readable to break it up into multiple statements or even another function to evaluate.

  114. 114
    Mark B. says:

    @👾 Martin:

    Load testing isolated systems is pretty easy. Systems like this which delegate a lot of the business logic ..

    This sounds pretty plausible. If that’s the real reason for the non-performant systems, they have a lot of work to do. In order to test the isolated system, you would need to create dummy connections to the external systems, and a sophisticated tester would try to create conditions where those connections would fail. But all you can do then is fail more gracefully.

  115. 115
    Emma says:

    @joes527: Yes. Phone companies that provide DSL services are just as bad. I had to get some information about my DSL service last week. By the time I got a live person — who couldn’t answer because why? — I was ready to rip the liver out of the next creature, two or four legged, that even looked at me funny.

  116. 116
    Gene108 says:

    @Steeplejack:

    And if you blame people for not automatically blaming Republicans on a host of issues, look at how many people here at “Obot Central” blame Obama/his Administration, when Sebellius has been publicly asking for private funds to fill in the gap HHS has with this roll out for the ACA.

    To think it may not have a trickle down effect on IT projects is wishful thinking.

    But when you run into a problem that is handled by the Executive branch the natural reaction is to wonder HTF did they screw it up.

  117. 117
    Gene108 says:

    @👾 Martin:

    One thing that was pointed out me is you have 50 states with their own IT systems, none of which are designed to work with other states.

    We are technically not rolling out a single Federal system, but 50 state systems.

    The Feds have to integrate a bunch if these systems because so many states refused to set up exchanges.

    It might be good thing in the long run, since uniform IT systems would help when we go to single payer.

  118. 118
    joes527 says:

    @Mike in NC: You seem to have missed the point. That wasn’t a joke.

  119. 119
    pseudonymous in nc says:

    FWIW, I think David Auerbach’s take is about right, which is that you have a disconnect between the frontend and backend systems, which taps into Martin’s comments upthread.

    And yeah, if that’s the deal, then it could be a long trek unless they get some authority to reinvent the wheel — that is, build some kind of ‘front backend’ that is web-scalable and pulls in stuff from those legacy systems ahead of time, then pushes back the transactions.

    Redditors claiming to know what’s up based on View Source can fuck right off.

  120. 120
    lol says:

    @Mark B.:

    There are a number of difficulties, mostly to do with the federal government hiring practices.

    As mentioned, the pay scale is lower than the private sector so it’s hard to attract talent.

    The best web developers are in New York City, San Francisco, etc, not in DC or other capitol cities and the Federal Government has generally made permanent telecommuting arrangements very difficult.

    The hiring process itself makes it hard to identify candidates with the necessary skills while weeding out the hacks and liars. And God help you if an utterly unqualified veteran makes it through the screening process.

    And that’s assuming you have budget to hire these people and the willpower to push through the red tape.

    In contrast, a contractor is pretty easy to bring on board.

  121. 121
    replicnt6 says:

    I’ve been a software engineer for > 30yrs (always in the private sector), and I have to say, I’m totally, wildly, unsurprised that they’re having problems. I’d be far more surprised if the sites ran smoothly.

    First off: c’mon dpm, do you really think there are government agencies with the amount of expertise and personnel required to build something like this? Of fucking _course_ they hired private contractors.

    It’s easy to forget that most of the massive web sites we use today didn’t start out massive. Commercial web sites generally follow a steady growth pattern (if they’re successful), starting from approximately zero. When your load grows steadily, so make periodic adjustments to handle the load. It’s pretty rare that a web site goes from zero to millions of hits literally overnight. And more often than not, when they do, they shit the bed. It’s hard.

    We (software engineers) collectively kind of suck at this. Perhaps one day this will really be an engineering discipline with accreditation. It’s kind of amazing that there isn’t, considering that pretty much every piece of technology these days requires substantial chunks of software.

  122. 122
    RSA says:

    @👾 Martin:

    The two people I know who are working on this tell me something a bit different…

    Thanks for the report. This makes a lot of sense and is consistent with the few larger-scale projects I’ve seen from a short distance.

    I’ll add one more observation–it’s easy to point at successful software projects in the private sector of comparable scope, and to ask why the government couldn’t have done the same. But as others have pointed out there are lots of unsuccessful commercial projects that fly under the radar because they were unsuccessful.

  123. 123
    jefft452 says:

    “The mercenary captains are either capable men or they are not; if they are, you cannot trust them, because they always aspire to their own greatness, either by oppressing you, who are their master, or others contrary to your intentions; but if the captain is not skilful, you are ruined in the usual way.” – Nicolo Machiavelli

  124. 124
    jefft452 says:

    1944 – US Navy “SeaBees” build a fully functional air base on a chunk of coral sticking a few feet out of the middle of the Pacific in 3 days, while under fire

    2004 – Kellog Brown and Root, after long delays and cost overruns, builds a barracks in the middle of a modern city, yet still cant provide electric lighting without electrocuting soldiers in the shower

  125. 125
    Steeplejack says:

    Just noticed that I bookmarked this a few days ago: “A Programmer’s Perspective on Healthcare.gov and ACA Marketplaces.”

  126. 126
    Mark B. says:

    There are a number of difficulties, mostly to do with the federal government hiring practices.

    Most of my experience is with state government, which has more flexible hiring practices than the feds. Nevertheless, it’s pretty uneven. Some state agencies do an excellent job, and others, not so good. I recently retired, so I can actually talk more freely about the job than I did while I worked there. Even in Texas, there was a lot a variance between agencies. Some did a really horrible job, but some, like where I worked, did pretty well. The pay issue isn’t as much as a problem as you would think, because the stability of the position and good benefits are pretty attractive to a lot of people in the IT field, especially some that have bounced around in a few failed startups in the past.

    Of course, we’re in Austin, so the local talent pool is pretty deep. It’s probably a lot harder to do as well in D.C.

  127. 127
    Lurking Canadian says:

    @Belafon: When I first discovered what lambdas were, I thought “Wow! Passing code as a parameter! Passing code as a *message*! Awesome!”

    Very quickly, I decided, “If I just wrap this code into a class that implements the appropriate Interface I will have (a) more readable code and (b) the ability to re-use it if I need it again”. I still think lambda functions are cool from the point of view of “Now how did they get *that* to work?”, but I am not convinced they are as necessary in as many cases as they are used in Java “how-to” guides.

  128. 128
    RSA says:

    @Lurking Canadian:

    I am not convinced they are as necessary in as many cases as they are used in Java “how-to” guides.

    I think it does depend on the language and the community of programmers who use it. Lisp (Scheme, CL, and maybe Clojure) functions, for example, tend to be written at a finer granularity, and lambdas/anonymous functions are indispensable.

  129. 129
    The Red Pen says:

    @Lurking Canadian:

    I am not convinced [lambda expressions] are as necessary in as many cases as they are used in Java “how-to” guides.

    You mean for Java 8? Up to Java 7, Java doesn’t have lambda expressions.

    I’m looking forward to them — that and extension methods. C# already supports both of these.

    When I say, “looking forward to them,” I mean I’m looking forward to using them myself. I cringe when I think about what a lot of other people are going to do with them.

  130. 130
    Uncle Cosmo says:

    Lies come from the contractor saying anything to win the contract, knowing that failure just means more money will be thrown at the problem.

    Ohhellyeah.

    About 30 years ago I had just ben hired by a major defense contractor & was assigned to a proposal for developing a prototype of a portable ruggedized computer system for maintenance and repair data. Our proposal manager was in his mid-20s, a Young Man To Watch in the company, & this was his first stab at winning a contract on his own.

    Our designers said that (given the technology of the time) there was no way to meet the specs without a CMOS chipset–regular less-pricey chips would simply overheat. So our design used CMOS–& we lost on cost to another major defense contractor that submitted a non-CMOS design.

    Some months later our PM went out to the program office for the delivery & demo of the first completed prototype.

    Within 10 minutes of power-on, the attache-size box was too hot to touch.

    Whereupon the winning contractor blithely requested (& got) additional funds from the program office for redoing the prototype using a CMOS chipset.

    When he got back to the office, steam still hissing from his ears, our YM2W said he would never, ever, make that mistake again…

  131. 131
    Lurking Canadian says:

    @The Red Pen: it was first in the context of Android programming that I encountered the concept, or something very like it. Turned up again in C#.

    I only write code “for fun” these days, so I’m not sure about versions and so on.

  132. 132
    Belafon says:

    @Lurking Canadian: Not everything needs an interface, especially when the interface is “take a string and a number,” like event handlers. Part of the reason I don’t like the “make an interface for everything” is this scenario: I have a dialog with three buttons and the dialog should respond to them. Since they are all buttons, they would use the same interface to respond. How does a class implement the same interface in three different ways? Instead, I can just pass the specific function to the specific button. There’s no need of an anonymous interface – no thanks Java, even though I am learning it – just to call a single function.

  133. 133
    Tone in DC says:

    @jefft452:

    1944 – US Navy “SeaBees” build a fully functional air base on a chunk of coral sticking a few feet out of the middle of the Pacific in 3 days, while under fire.

    2004 – Kellog Brown and Root, after long delays and cost overruns, builds a barracks in the middle of a modern city, yet still cant provide electric lighting without electrocuting soldiers in the shower.

    Shaking my head. Halliburton and KBR are the pinnacle of Beltway Bandit ignominy. Remember when Cheney’s company provided the troops in Iraq VERY expensive water, which had fecal matter in it.

  134. 134
    mclaren says:

    No, mistermix, you have gotten it wrong in every possible way. As mentioned by the first commenter, there is no “bottomless trough of pork” in software contracts awarded by the federal government.

    The only bottomless troughs of pork I know of in the federal budget are 1) no-bid contracts awarded to weapons and other military contractors, which are basically a form of bribery — as witness the no-bid contracts awarded to Halliburton during the Drunk Driving C Student’s reign of error because coincidentally the Vice Torturer (excuse me, Vice President) Dick Cheney just happened to be the former CEO of Halliburton; and 2) no-bid contracts awarded to former military and executive branch personnel who cash in by starting security-related companies. The best example here is the worthless explosive-sniffer units built and sold for billions to the federal government by a company started by the first head of the Department of Homeland Security. The explosive sniffers worked so poorly that after buying them, the federal government stopped using ’em and wrapped ’em up in plastic sheeting and stored ’em in a warehouse.

    No, the real reason for this gigantic clusterfuck in software development, mistermix, was enumerated by Fred Brooks in his classic book The Mythical Man Month: Essays On Software Engineering.

    Fact: 20% (1 out of every 5) of all large software projects end in failure.

    From the IEEE Transactions article “Why Software Fails”:

    Last October, for instance, the giant British food retailer J Sainsbury had to write off its US $526 million investment in an automated supply-chain management system. Merchandise was stuck in the company’s depots and warehouses and was not getting through to many of its stores. Sainsbury was forced to hire about 3000 additional clerks to stock its shelves manually.

    This is only one of the latest in a long, dismal history of [software] projects gone awry. Most IT experts agree that such failures occur far more often than they should. What’s more, the failures are universally unprejudiced: they happen in every country; to large companies and small; in commercial, nonprofit, and governmental organizations; and without regard to status or reputation. The business and societal costs of these failures — in terms of wasted taxpayer and shareholder dollars as well as investments that can’t be made — are now well into the billions of dollars a year.

    The problem only gets worse as IT grows ubiquitous. This year, organizations and governments will spend an estimated $1 trillion on IT hardware, software, and services worldwide. Of the IT projects that are initiated, from 5 to 15 percent will be abandoned before or shortly after delivery as hopelessly inadequate. Many others will arrive late and over budget or require massive reworking. Few IT projects, in other words, truly succeed.

    Yes, you read that correctly. 1 out of every 5 big software projects get 5 or 10 years in, the software doesn’t work, no one can fix it, and the project gets cancelled after millions or billions of dollar get pissed away with nothing to show for it. One of the most beautiful examples is the massive 208 million dollar upgrade to the State of California’s DMV software. After 6 years of work, the project was abandoned because no one could get the software to work.

    Programming is a con job operated by ignorant incompetent grifters. Software is garbage and most software doesn’t work and can’t be made to work. The primary reason for this is obvious — because programmers constantly reinvent the wheel, changing their programming languages every week or so. From FORTRAN to C to C++ to C# to Ruby on Rails to ADA to [fill in the blank], the game of software musical chairs never ends. The purpose? Equally obvious: to keep programmers in job security. No one wants to admit this, any more than anyone wants to admit that chiropractors are con artists who do nothing to improve anyone’s health, that the Meyer-Briggs “personality test” is pure pseudoscience without the slightest grounding any kind of empirical science, or that hair and fiber “evidence” used in criminal trials is pure guesswork distinguishable from dowsing only by the terminology employed.

    For years, forensic hair examiners have testified that physical characteristics of hairs can be identified and used to establish the presence, or absence, of certain people at a crime scene.

    The problem? No scientifically-accepted statistics exist about the frequency with which particular characteristics of hair are distributed in the population, according to the NAS. And there appears to be no uniform standards on the number of matching features that must be present for an examiner to declare a match.

    Recent studies reveal that microscopic hair analysis is not yet a precise science. One study by the FBI, cited in the NAS report, found that of 80 hair comparisons done through microscopic examinations, 9 of them, about 12.5 percent, were found to come from different sources when reexamined through DNA analysis.

    Source: “Forensic Tools: What’s Reliable and What’s Not So Scientific” from the PBS website for “The Real CSI”.

    tl;dr — much of the so-called “work” done by average Americans in everyday life in 2013 consists of various con jobs and pseudoscience disguised as valid disciplines. Programming is one. But there are many many more, and no one wants to admit this.

  135. 135
    Belafon says:

    @mclaren: Love how you showed up late.

    The primary reason for this is obvious — because programmers constantly reinvent the wheel, changing their programming languages every week or so. From FORTRAN to C to C++ to C# to Ruby on Rails to ADA to [fill in the blank], the game of software musical chairs never ends.

    In my 15 years of professional programming, I have used two languages primarily, C++ and C#. I use Python for small tasks.

  136. 136
    mclaren says:

    @Mark B.:

    Thank you for providing us with a perfect example of a typical arrogant ignorant incompetent programmer explaining how “simple” everything is — yet who typically can’t write a piece of code to so something as simple as accept a name and file it in a database reliably.

    Yeah, you can’t just do an ‘out of the box’ test with a standard product, but really building a script that does things like transactions and verifies the results isn’t all that hard.

    Let’s debunk your first startlingly ignorant statement: a transaction in any computer system typically requires entering a name, and as wiser and more skilled programmers than I have repeatedly pointed out, merely accepting a name and entering it correctly into a database is a task which no programming language and no programmer has yet managed to accomplish.

    There’s a classic essay with which you’re obviously too ignorant and too incompetent to be familiar which describes the manifold problems which beset software “engineers” when they try to write a program that accepts a name. Either the software or the programmer always makes invalid assumptions: that a name contains no numbers, that a name must have a first and last name, that a name cannot be more than 256 characters in length, that a name will contain only printable ASCII characters, and on and on and on.

    The essay is called “Falsehoods programmers believe about names.” You should read it. Of course you won’t, since you’re not only ignorant and incompetent, you’re also too arrogant to realize how ignorant and incompetent you are. A classic example of the Dunning-Kruger Effect.

    Yeah, you won’t handle all of the edge cases, but you should be able to generate a realistic amount of load using the normal business process without too much effort. Even if you’re not using one of the common test suites, creating web requests and listening to the responses is pretty basic stuff.

    Yes, it’s all “pretty basic stuff.” That’s why 1 out of 5 of every large software projects crashes and burns, gets abandoned without ever getting to the point where it works. Because it’s “pretty basic stuff.”

    A more realistic view of programming comes from the essay “Why can’t programmers program.”

    Hint: They’re talking about you, Mark H.

    Yeah, you’re still going to sometimes have weird problems in the real world which your testing didn’t anticipate, but you need to do your due diligence and test as much as you can. I don’t see much evidence this was done here.

    Because if you just do your homework and “due diligence” and “test as much as you can” you can anticipate most of the real-world problems. Except of course that you can’t. Because of the combinatorial explosion, of which you’re far too ignorant to be aware. The number of possible combinations of inputs in any even modest-sized software system explodes as the factorial of the number of inputs, which so rapidly grows beyond the bounds of the number of elementary particles in the universe as to be completely and utterly untestable.

    But of course you’re too ignorant and too incompetent to realize any of this, Mark B. A typical programmer.

  137. 137
    Manyakitty says:

    @Kay: As of this morning, it still won’t recognize my ID/password so I can’t even log in. Has anyone made it through yet?

  138. 138
    Mark B. says:

    They’re talking about you, Mark H.

    Why can’t arrogant commenters spell?

    Yeah, I’m familiar with codepages, character sets, unicode and all of the difficulties related to translating all of that. If you read the whole thread, you might have seen that I dialed things back when someone mentioned that the problems were probably due to interaction with legacy backend systems, which is a pretty intractable problem. But I stand by the rest of my post. All of the issues you mention are solvable and manageable if you design the product correctly. I didn’t say it was easy.

  139. 139
    Mark B. says:

    @mclaren: It’s interesting you mention names, one of the problems that I tackled in the last couple of years was a name interpreter for election results. It was an interesting problem that requires lots of human interaction to correctly interpret the names, but again, not really insolvable.
    ]
    P.S. Dude, you should dial back the anger. Not really good for you health.

  140. 140
    Mark B. says:

    @mclaren: And sorry to reply so many times to your post, but I have my own theory why many large software projects fail. It’s because the upper level managers are dolts that don’t know much about software, and most programmers are pretty myopic about solving very focused problems instead of figuring out how things are supposed to fit together. Complexity is manageable, but you sometimes have to simplify the requirements to make it so, and not all managers are willing to do that. Things like restricting the user to a constrained sequence of interaction with the UI can be helpful, although it’s not the most user-friendly interface.

    Really, I have a degree in mathematics, I kind of know the theory.

  141. 141
    Pococurante says:

    In a nutshell, here are my educated guesses of why exchange software is failing:

    I realize you are a non-Subject Matter Expert blogger on a blog devoted to bloviating, but how about next time instead of pulling poo out of your shorts and proclaiming it poo, maybe you could be troubled to actually educate your opinion.

    Comparative exchanges are based on the quality of the system integration points. Those points are controlled by other bodies.

    Those other bodies have been given every reason to delay and distort.

    To phrase it another way, any fool can throw up a dam in front of their river and blame folks downstream.

    And any damn fool can whine about it with smokescreen arguments; witness you.

    Stick to making stupid “me too” comments about some B2C’s latest consumer device gaffe.

    I’ve been integrating B2E/B2B/B2C common exchanges since the mid 1990s. You’re an embarrassment trying to paint a difficult problem onto contractors. False equivalence with a bullshit tinge of high techno-geek in a white jacket.

Comments are closed.