woah that was quick pic.twitter.com/xt9IWCip5j
— Joe Groff (@jckarter) October 4, 2021
Is Facebook down right now? This global outage is tough on everyone https://t.co/YLuhCc5Pca pic.twitter.com/HVis0HnR1I
— The Verge (@verge) October 4, 2021
The Verge says it’s back online now, so don’t panic…
Just as Facebook’s Antigone Davis was live on CNBC defending the company over a whistleblower’s accusations and its handling of research data suggesting Instagram is harmful to teens, its entire network of services suddenly went offline.
The outage started just before noon ET and took nearly six hours before it was resolved. This is the worst outage for Facebook since a 2019 incident took its site offline for more than 24 hours, as the downtime hit hardest on the small businesses and creators who rely on these services for their income.
After failing all tests for most of the day, a test of ISP DNS servers via DNSchecker.org showed most of them successfully finding a route to Facebook.com at 5:30PM ET. A few minutes later, we were able to start using Facebook and Instagram normally, however, it may take time for the DNS fixes to reach everyone…
Inside Facebook, the outage has broken nearly all of the internal systems employees use to communicate and work. Several employees told The Verge they resorted to talking through their work-provided Outlook email accounts, though employees can’t receive emails from external addresses. Employees who were logged into work tools such as Google Docs and Zoom before the outage can still use those, but any employee who needs to login with their work email was blocked…
So if you ever think you’re having a rough Monday… just know it can’t be as bad as working for Facebook right now ☠️ https://t.co/AItvqQCTgR
— Marques Brownlee (@MKBHD) October 4, 2021
According to Facebook, the only way to fix the FB outage is to cover its servers in horse paste.
— Kashana (@kashanacauley) October 4, 2021
And here *I* thought the Pandora Papers dump would cause more disruption sooner than last night’s 60 Minutes expose…
There is an entire generation of political staff who are dealing every day with how this algorithm is literally inspiring people to violence and blind rage against elected officials. And we are all fed up with FB officials excusing or spinning this very real problem. https://t.co/9G0VRZJpaS
— Anne Caprara (@anacaprana) October 4, 2021
A whistleblower claims Facebook prematurely turned off safeguards designed to thwart misinformation in a moneymaking move that contributed to the deadly invasion of the U.S. Capitol. Former Facebook product manager Frances Haugen spoke on “60 Minutes.” https://t.co/EvZJJxlRvh
— The Associated Press (@AP) October 4, 2021
ok heres page with all the clipshttps://t.co/u0SPMxHmgA
— rat king (@MikeIsaac) October 3, 2021
If you’re also wondering how the they pulled this all off so we all could know the truth, it’s because of (a) her bravery and (b) support from the incredible @wbaidlaw, which helps protect whistleblowers to get their stories out. And they need donations: https://t.co/9cTpKkkas6 https://t.co/JuvE3WgdHH
— Ian Bassin (@ianbassin) October 4, 2021
BC in Illinois
Here’s the official response from Facebook spokesperson, Blaire Erskine.
Baud
I felt a great disturbance in the Net, as if millions of voices suddenly cried out in terror and were suddenly silenced.
Gin & Tonic
Pretty much everyone who can understand the technical explanation of what happened has a horror story in their past of an “oh, shit” moment similar to this. It’s just that they probably didn’t fuck up a company with 60,000 employees and a billion users.
HeleninEire
@BC in Illinois: She is so smart and funny.
ETA: why isn’t she famous outside of the internet?
Baud
Looking forward to when Bitcoin goes down.
jl
Didn’t watch 60 minutes report, but read summary of her charges. Facebook’s response was such a hash of non-denial denials, misdirection and strawmen, that it greatly increased my confidence in Haugen. Facebook is still trying to conflate the freedom of quirky old Aunt Bea and Uncle Fred to spout whatever as individual users, with Facebook’s freedom to drive traffic with their algorithms to segment users by various demographics and maximize ad revenue. Facebook using secret sauce AI to manipulate its membership doesn’t even seem to rise to a corporation’s right to commercial free speech to me, but any BJ lawyers around, let me know if I’m wrong there.
Benw
When my connection to my work servers go down I’m like dude take the day off and go to the beach
OzarkHillbilly
Nice to know I’m nobody.
Baud
@OzarkHillbilly:
C’mon. You knew that before today.
HumboldtBlue
I haven’t logged into Facebook since 2015 when family members traveled to Ireland and they wanted to taunt me with pics.
jl
Coincidentally, I found this story at Barry Ritholtz’ The Big Picture blog this morning. I’d be interested to hear what people who are interested think of it. Seems to connect Facebook’s ongoing carnival of crime, swindles, dishonesty, and malfeasance to inability to maintain market share, and maybe more importantly, maintain engagement, page views and time, and clicks among the old fogeys who remain. But, its business model requires an ever increasing malignant rate of growth, so even maintenance isn’t enough
Facebook Is Weaker Than We Knew
A trove of leaked documents, published by The Wall Street Journal, hints at a company whose best days are behind it.
Kevin Roose, NYT
https://www.nytimes.com/2021/10/04/technology/facebook-files.html
Note: edited in attempt to eliminate traces of depraved apostrophe crimes against humanity.
Steeplejack
I use Facebook hardly at all—mainly to passively receive notifications from a handful of people—and I was mildly surprised to find out how far-reaching its collapse was. I hadn’t thought of people using their Facebook identity as a log-in to various sites, not to mention businesses basically operating on line via Facebook.
Steeplejack
@Benw:
Even people at Facebook were calling it a “snow day”—on other platforms. ?
prostratedragon
@Baud: Just before the end they all shouted but one word:
FEWER!!!
Sure Lurkalot
@Benw: Ha! One job I had everything went down because a squirrel committed suicide via power pole. Cheapskate boss was out of town and after 2 hours and reports that repairs were going slowly, the head of HR told everyone to go home. 2 hours later, things were repaired and cheapskate boss had HR call everyone to come back. At like 2pm.
schrodingers_cat
BJP spreads its poison through WhatsApp. It is its most diabolical and successful outreach tool. And it has made many formerly smart people into blithering idiots who mindlessly forward BJP propaganda after inhaling it themselves. So I am glad that it is down at least for sometime.
Facebook India has had a thumb on the scale for the BJP and has helped spread the poison of hate and bigotry.
jl
@jl: Actually, the Facebook AI to segment its audience isn’t secret sauce. It’s easy to find everything from introductory explanations, with step-by-step guides on how to game it, to deep weeds tours through the math.
If Mr. Facebook doesn’t know what his company is doing it’s because he doesn’t want to. He has many many people who can explain the math to him.
Note: also edited to conceal traces of apostrophe crimes against humanity.
Another Scott
@jl: No click from me, but isn’t the headline old news? There were jokes a year or few ago that the young hep cats regarded FB as AOL or something and they were off on TikTok and whatever else now. I’m sure that just like Gates and MS before the “internet tidal wave”, they’re doing everything they can to get their near monopoly mojo back…
Cheers,
Scott.
jl
@Another Scott: I kind of agree. The article goes through the parade of horribles and ties it in to Facebook’s increasing difficulty in making its business model work and stock valuation plausible.
So, YMMV may vary on what the article does.
Edit: in particular, increasing need to do everything ever cheaper, so no due diligence in trying to get adequate staffing to do what Facebook says it will do, or is doing, or has tried to do in order to not be a malfeasant lying corporation.
HumboldtBlue
Here’s a really cool clip of a drone flying over a lava flow and it didn’t come from Facebook.
schrodingers_cat
@Another Scott: Teenagers describe FB as something that their grandparents use.
Martin
For those who aren’t technically inclined, the outage is more or less the equivalent of being the only guy who can lock up the bank vault at night, and accidentally locking yourself into the vault with no way to unlock it from the inside.
It’s one of those tasks that can’t have safeguards apart from extraordinarily detailed checklists that you always have two people implement in very formal ways, especially if you’re the size of Facebook. Someone really fucked up ‘following directions 101’ on this. Imagine guys in missile silos calling out and validating codes and turning keys simultaneously. That kind of shit.
Another Scott
@HumboldtBlue: Original source is Instagram (part of FB)… ;-)
Thanks!
Cheers,
Scott.
Anoniminous
@jl:
Too late. We have our I’s on you.
Roger Moore
@Gin & Tonic:
This kind of thing is why older companies like IBM- and now Microsoft- can be slow moving. They’ve been bitten by problems like this before, and they’ve survived by adding a procedure to prevent it from happening again. By the time they follow every procedure to prevent every bad thing they’ve ever encountered, it can slow them down a fair bit. OTOH, they don’t wind up dealing with this kind of problem nearly as often as the newer companies that haven’t implemented every safeguard.
Martin
@schrodingers_cat: My daughter was unaware that they were all down. My wife noticed immediately.
HumboldtBlue
@Martin:
Thanks.
Now can you explain to me how there is a weather delay in a stadium that has a roof? And just what the hell is going on with L,A. weather?
raven
@schrodingers_cat: A good friend died in Tucson last night and a bunch of us who are dispersed all over the world have been taking about him on Facebook. Yea, we’re old.
jl
@Anoniminous: You guy’s will never catch me. Its impossible.
Benw
@Steeplejack: kids, and maybe Facebook programmers, learn more about having a life on snow days than any given week of school/work
Anoniminous
@HumboldtBlue:
Lightening
Why a domed stadium is unsafe during a lightening storm is beyond my pay grade.
Roger Moore
@jl:
The thing that gets me is just how much Facebook has just plain lied to steal people’s money. Their whole advertising analytics is one lie after another. They lie about click through rates, they lied about the importance of switching to video, and so forth. It’s probably easier to find things they didn’t lie about. And every one of those lies has left a wake of companies that failed because they listened. The whole Facebook C-suite could spend the rest of their lives in prison, and it wouldn’t come close to providing justice for everyone they’ve screwed.
HumboldtBlue
Speaking of you olds, Dick Tracy appeared for the first time on Oct. 4 1931.
schrodingers_cat
@raven: Hey I am no spring chicken either and certainly no teenager.
BTW I am on WhatsApp because of connections in India but not on Facebook.
jl
@Benw: One thing in the article I linked to has that I haven’t seen before are details of Facebook’s desperate and hilariously sad pitches to try get more young people and kids. Which I found to be funny like a clown.
Martin
@Roger Moore: Except DNS is everything for Facebook. They don’t exist if it’s down or wrong. So you build procedures around it that you never, ever deviate from. It needn’t be slow, but that discipline has to be there. I know one of the guys that does it for my employer and there’s a minimum number of people who need to be present before they update the master DNS. You add speed by adding redundancy of staff so that someone out sick doesn’t force you to either delay to the next day or deviate from procedure.
Roger Moore
@schrodingers_cat:
Facebook helped facilitate the Rohingya genocide in Myanmar. These are awful people, and they deserve to spend the rest of their lives in prison.
Anoniminous
@Roger Moore:
If the computer industry was held to the same standards of professional accountability of, say, the average car mechanic every one of the buggers would be bankrupt.
Benw
@Sure Lurkalot: I’m sure productivity soared that afternoon.
A foolish consistency is the hobgoblin of small minds.
schrodingers_cat
@Roger Moore: They played and are playing a big part in spreading BJP’s poison. They helped orchestrate last year’s pogrom in Delhi among other things.
Roger Moore
@Martin:
My point is that companies like IBM and Microsoft have managed to become venerable by the standards of their industry by creating a kind of corporate memory. That has a good side- they don’t screw up important things like their company DNS- and bad sides- they can spend so much time following checklists, they aren’t as nimble as the newcomers. Of course nimbleness can be massively overrated. It does you no good at all to be supremely nimble if it just lets you get in trouble faster.
raven
@schrodingers_cat: From a good buddy on FB
NotMax
Suppose someone should take note that it’s the first Monday in October, so SCOTUS is officially back in session.
Fasten your seat belts, it’s going to be a bumpy term.
Baud
Reminds me of when Cole forgot to renew the URL for this place.
Roger Moore
@Anoniminous:
If the computer industry were held to the same standards as the average car mechanic, they would have developed completely differently. Computer code can be made to be safe- those cars depend critically on computers, after all- but it requires a degree of care that most computer people have never had to deal with.
It’s actually interesting to hear computer people talking about the use of computers in safety-conscious industries. They just don’t seem to get the background of car and medical device manufacturers, so they don’t understand why they don’t want to run their computer development the same way.
PJ
If only Facebook could be down forever, it would be one step in stemming the cancer that is killing the free world. I don’t think our lawmakers have the guts to do it, but a boy can still dream.
Gin & Tonic
@Martin: But this wasn’t really a DNS problem, it was a BGP problem. Even with checklists, BGP is easy to fuck up.
schrodingers_cat
@raven: I am not personally afraid of FB but do worry about their malign influence on democracies. They need to be reigned in. Most of the anti-vax nonsense is coming from a few FB account. They have weaponized stupidity and bigotry.
Gin & Tonic
@Roger Moore: If it’s true what was reported, that FB’s physical access control system (card-key access to doors, for those who don’t deal with this) was reliant on Internet-facing DNS, then that’s just a badly-designed system that no amount of checklists and procedures can cure.
dmsilev
@Roger Moore:
Tesla, which is basically ‘what if we ran a car company like a computer company’, rolling out a beta version of their self-driving car software to end-users being a case in point. What could possibly go wrong?
Martin
@HumboldtBlue: Yeah, it’s pretty weird. 4 thunderstorms so far this year. Probably been 5 years since we’ve had the last one.
Tony Gerace
@Martin: Either … 1) The speculation about a massive database cleanup today are true. … or … 2) The design and implementation of Facebook is deeply incompetent. … or … both I used to work in a group that supported online systems in a hospital. An unplanned 8-hour outage? All of our asses would have been fired.
raven
@HumboldtBlue:
raven
L85NJGT
FB had exclusivity with a commercially desired user base early on, but has marketed for shit ever since, and morphed into the online MeTV. That’s on Zuckerberg.
They just start offering hosting services (aka “cloud”) last year. Good luck playing catch up in that market, and we just landed back on the visionary with no vision.
FB will end up going the aol & yahoo route, squeezing a few bucks out of a residual user base.
Gin & Tonic
For anyone who is interested, here is a lengthy explanation of what appears to have happened with FB et. al. today.
Shakti
Anyone familiar with NC-09 or Charles Graham? I love this ad, btw:
Mary G
@HumboldtBlue: We had huge thunder and drizzle! It was dripping off the roof, even though the driveway didn’t all get wet. I’m in that stage of drought when 0.01 inch of rain is celebration-worthy.
Our June gloom weather has lasted all summer. I can’t count the number of times I looked at my phone and it was 73 degrees. The highest temp inside my house was 84 degrees all summer. We’ll probably get hellish weeks of 100+ degree weather with no humidity and Santa Ana winds bearing smoke and fire all autumn to make up for it.
Dan B
@Martin: Seattle has very little lightning, probably an average of four bolts a year and never in summer, until this year. Where I grew up in Ohio and Arkansas many homes had lightning rods. It was common to see trees blown apart by lightning (also cars, statues, and building bits in fields – but that was twisters). I was surprised that apartment buildings in Seattle had no lightning rods. Much of California seems the same.
Anoniminous
@Roger Moore:
I found it interesting in 1985 when I was working in SillyCon Valley. In 2021 the unceasing Amateur Hour is just tiresome.
People are dying because TechBros can’t act like grown-ups.
Jim, Foolish Literalist
@Shakti: Just saw that. Looks like he’s got a steep climb– R+6 district, I’m not familiar with the incumbent R, Dan Bishop, but I’m glad to see a Dem using 1/6 in a campaign .
Dan B
@Gin & Tonic: I Jeff Tieldrich (sp?) Tweeted: Instagram is down. Can I come over and look at your food?
Not technical but droll and to the point. Andy Borowitz quoted Facebook saying they realized they helped organize the attack on the congressmen they own.
West of the Rockies
Can’t say I feel any schadenfreude over this. I don’t see FB or Zuckerpig especially suffering in any meaningful way. It’s more like if professional wrestling got canceled. Don’t watch it, wouldn’t care.
Another Scott
@Gin & Tonic: Thanks for the pointer.
Cheers,
Scott.
gene108
@schrodingers_cat:
India is WhatsApp’s largest market by far. I have WhatsApp because all my relatives in India use it. It’s ubiquitous there.
What’s not to like about free calls, free video calls, and messaging people anywhere in the world?
Jim, Foolish Literalist
I wish him luck, if nothing else it will cost Lee money
different-church-lady
At long last Facebook is learning how perilous it is to rely on Facebook for every goddamned thing.
schrodingers_cat
@gene108: It is the vehicle by which BJP injects its poison directly in the veins of Indians who uncritically forward garbage generated by the infamous IT cell.
Other than that it is wonderful. And I have used to call and message my family in India and be a fact checking killjoy when they forward me some baloney about Sanskrit scriptures and their cult leader Modi
I get few if any forwards now.
Barbara
@L85NJGT: “An online MeTV” sounds about right. I use it occasionally keep track of political groups that use it as a way to communicate events. My fitness studio does this as well for feedback on classes. But I find that most people I know posting on Facebook for personal stuff are just wasting time. It’s mostly inane once you’ve seen enough cute puppy videos. And I tuned out completely on my cousins who use it to fight about politics. Their posts are not all that coherent even when I agree with them.
Two of my three kids absolutely disdain it. The third (coincidentally the oldest) uses it to blow off steam. I think she should find a better outlet.
Bill Arnold
Got a Facebook recruitment email in August (2021), which included this sentence, bold mine:
I might have responded, if that part in bold hadn’t been present. (Though almost certainly with a polite no, happy with my current employer.) The part in bold actually made me angry for a moment.
Facebook fucked around for many years. Now they are finding out. And Zuckerberg even made a bet on Trump. Very not smart.
Enhanced Voting Techniques
@jl: From what I’ve scene, Facebook’s businesses model is just spamming you with the someone replied to your post comments to suck you into never ending flame wars.
Ken
It depends. Did they chat with their[*] congresspeople and get the necessary fixes to the liability laws pushed through, such that a self-driving car that kills a pedestrian is just one of those “whoops” things, no fault here, too bad so sad?
[*] English possessive pronouns have so many meanings. Obviously I mean the congresspeople who represent the Tesla corporation, not the ones owned by the Tesla corporation.
Shakti
@schrodingers_cat: Same. I’ve had no success in getting my relatives onto an alternative platform like Signal. But weirdly, my OLD contacts keep popping up.
It seems like the same relatives who used to send chain letters and email forwards used FaceBook as adult tumblr and use WhatsApp to send dumb inflammatory memes. My phone cries out every day for me to delete the memes and videos from the family group. They will not stop.
dmsilev
@Ken: As I understand it, somewhere in the piles of legalese that the actual suckers, I mean customers ‘agree’ to before being able to use the software is a statement that said sucker acknowledges being in ultimate control of the vehicle and has all responsibility in case it, let’s say, rams a set of police cars parked on the side of a highway with their flashers on.
To pick a completely random and I’m sure totally hypothetical scenario.
karen marie
@schrodingers_cat: Excrpt for logging in to games. I ppay a game that requires me to have FB account, so I do, although it’s a fake name with a burner email address and completely locked down.
It really irks me but there doesn’t seem to be any way to force game companies to change to something else.
Jim, Foolish Literalist
@Shakti:
HumboldtBlue
@Mary G:
Yeah you will.
opiejeanne
@schrodingers_cat: monstrous.
HumboldtBlue
@raven:
Thanks
Major Major Major Major
Legit didn’t notice it went down, lol. Until everybody was chatting about it on work slack.
Major Major Major Major
Oh, and my grandpa might be dying and my aunt has indicated she doesn’t plan to honor his DNR.
Barbara
@Major Major Major Major:Very sorry about your granddad.
Urza
@Dan B: Greater Seattle at least has had thunderstorms the last 3 summers. I was told it hadn’t happened since the 90s before that.
HumboldtBlue
@Major Major Major Major:
Oh no, that’s terrible. Here’s hoping for the best.
Dan B
@Urza: I couldn’t remember if there were summer thunderstorms last year or the year before. It seemed like last year and this year were active. I seem to recall T-storms coming from the east iver the Cascades but this year they originated in the Puget Sound lowlands.
The heat waves have been very disturbing. The city is not ready and locals seem to believe they’re an aberration that won’t happen again instead of a sign of a new trend.
Starboard Tack
I once accidentally unplugged a server in the middle of a 6 hour backup. The software was so old it wouldn’t continue from the interruption. My fuck up made the cheapskates upgrade the app to one that took less than an hour. I did not get thanked, but also not fired.
Another Scott
In other news, … TheHill – 1.8% of employees fired for refusing vaccine at Northwell NY health system. They phrase it a little differently, of course…
To put that number in context, nationally quits have been rising all year and are above 4M a month now.
The employed civilian labor force is around 153M, so 4/153 = 2.6%.
An organization that large probably has over 1.8% annual turnover anyway…
I’m sure this is bad news for Democrats and Biden.
Cheers,
Scott.
Another Scott
Cheers,
Scott.
Chris Johnson
@Roger Moore:
“Move fast and break”