I’m sure Adam will have a much more complete post about the Nashville bombing as soon as facts are more clear. In the meantime, I just want to note that the bomber sure picked an effective target. His apparent intentional and careful targeting of what the Post is calling the AT&T “transmission building” led to a regional telecommunications blackout of pretty impressive proportions. 911, cell and Internet service were affected. The Nashville airport closed for around four hours.
A few years ago I toured a similar facility in Rochester, and it’s an impressive building. It is overbuilt — something like 4 stories tall, but built so heavily that it could have been ~15 (my memory isn’t 100% on that) — lots of brick and concrete. It has redundant generators, steel roll down window shields in the lobby, and 24/7 security. It has to be impressive, because, like the Nashville building, most of the Internet and telephone traffic for the region passed through it (at least at the time I toured it).
I’m not revealing any secrets when I say that every city has a similar building, they’re a pretty important failure point, and they’re generally downtown, facing busy city streets. They probably should be better protected, but I guess we needed to spend our post-9/11 money on turning cops into stormtroopers and giving them armored vehicles instead of hardening these buildings. Another factor must be the almost complete lack of regulation of telecoms once they branched out from land lines into cellular and Internet.
Immanentize
OT, but funny:
NotMax
So, infrastructure weak?
:)
Don’t go blaming the victim(s).
raven
It’s infrastructure week!
Flooding forces early voting site at Athens-Clarke Extension Office to close
Immanentize
On topic, ATT has been a target for many reasons over the years. One of the most consequential cases ended up being Abbate v. United States in SCOTUS.
Two or three trade unionists, trying to unionize AT&T in the 50s, travelled from Chicago to Tennessee to blow up some transmission buildings. They were caught and tried in Chicago to a (likely) pro-union jury and judge and we’re sentence to — six months. The US (and the President’s Analyst) were not happy with this and tried them in federal court for the same criminal acts and they received multi-year sentences.
This gave us the “dual sovereignty doctrine” which survives til this day that allows both State and Federal prosecutions for the same acts — think drug deals these days.
Immanentize
@NotMax: The new suck building in Boston — we all got to watch it go up while they were finishing the big dig. It was made plane crash proof, I believe. More thicker concrete than the Maginot Line.
RobertDSC-Work
There is one just down the street from my workplace.
I laugh when I see the PacBell logo on one of the signs on a side fence. I know what that logo looks like, but a lot of people may not.
Brachiator
@Immanentize:
There is some heavy irony deployed here.
Leto
I’m trying to think of who would actually pay to move all this infrastructure. You think the military budget is large? Can’t wait for people to actually see the cost involved with 1) buying land out in the middle of nowhere 2) building an entire facility to military grade security standards and 3) rerouting all the fiber/copper to these “secure” points. Of course we also have the security force needed to provide 24/7 protection: men, sensors, and other equipment.
At my last base I was the lead project manager for the construction effort for our new communications facility. This involved 1) the building and 2) moving 2 of the 8 transatlantic fiber cables to our facility. This would be a similar project but multiplied, maybe, a 100 times over? More/less? This should be fun.
Stevie
I used to work in one of these buildings. You just made me realize. When I was in college I worked in a call center for a mail order fruit basket company during the holidays. It’s in a big brick building downtown that belongs to Qwest now.
NotMax
@Immanentize
New (at the time) library at one of the alma maters was designed and built during and in the aftermath of the later 60s and an ongoing time of student unrest. Could stand in as a stolid medieval fortress in any film, recessed windows look like arrow slits.
Brachiator
Regulation ain’t magic.
Leto
@Brachiator: one of the biggest complaints against legislators is that they don’t understand the technology they’re trying to regulate. Which if you look at the past twenty years worth of internet regulation, is a fairly spot on critique. Either they don’t understand it, or the people they hire to help craft legislation for them don’t. And that’s not even getting to who we put in place to run the FCC.
zeecube
There’s one in my town. About 4 stories. Encircled by 10 foot iron fence. No windows – just narrow slits on each side where you can stick a gun out to shoot at the riffraff trying to storm the building.
artem1s
has it been determined that ATT was the target? my understanding is this was a suicide and the place was randomly picked. maybe the lockdown was his beef and COVID; this was a high tourist area. maybe the timer was programmed incorrectly and it was supposed to go off right as the bars/restaurants let off for the night. the RV was parked there at 1:30 AM. they don’t know if the driver was still alive when the bomb exploded. sure it revealed a weakness in the infrastructure but it caused a temporary regional shut down of mobile calls and 911 – an actual emergency with thousands making mobile calls all at the same time can cause the same problem.
L85NJGT
The reason these buildings sit in central business districts is…. ahhhh… demand for service. Running separate cables from every office in every building to an undisclosed location would be stupidly expensive.
A 4-8 hour (partial) outage just isn’t worth the buildout costs.
japa21
@artem1s: As I mentioned last night, the RW considers this a direct shot at AT&T.
Sure it is just BS, but this is what they believe.
Brachiator
@Leto:
They regularly demonstrate this in public hearings. And to be fair, or more damning, their staff often are just as bad, as you note.
This problem might be compounded by the fact that a lot of tech leaders appear to be kinda “techno libertarians” who believe that you should just take a hands off approach to technology and innovation. This has led to tech people essentially withholding their expertise when it comes to some issues related to regulation, even when they privately agree that there are real problems that should be addressed.
There is also an attitude among some of them that they are technology high priests, above the concerns of the “norms.”
You often see the range of positive and negative attitudes expressed by guest on various tech shows, such as This Week in Tech on the TWIT network, hosted by Leo Laporte and other similar shows.
As a side note, Congress is also thrown off by the fact that the tech industry does not give them as much in campaign contributions as other industries, proportionally speaking. And does not ask for favors to the same degree. As a counterpoint, stuffed into one of the recent relief bills are some favors for the film industry, I believe.
The Moar You Know
Not one cent, even of other people’s money, will be plowed back into existing telecom infrastructure, not until every dime of their expenses is paid for by taxpayers and they are granted a complete liability shield.
For a great example, look no further than California’s own SDG&E and PG&E. Not one dime for improvements – and they’ve said this overtly – until all their expenses are paid by the public (including any improvements) and they’re immune from all lawsuits. This is the model going forward for all utilities.
NotMax
@Leto
Truisms:
1) The [furnace/refrigerator/washer/insert major appliance of choice} will crap out on weekends or holidays.
2) Telecommunication legislation will lag a minimum of one generation behind technological advance.
L85NJGT
@Brachiator:
Local services are tariffed and uptime requirements are monitored but there wasn’t enough downtime to even get customers refunds.
Functionally this was the equivalent of a jackass with a backhoe.
J R in WV
The AT&T switch building in CRW — Charleston, WV, is at least 14 stories high, in a community of around 46,000 Also state capital, Kanawha county seat. V few windows of any sort, I counted stories/windows in an older bank building next door to estimate it’s height. Big generator set on the roof, CAT engine, lots of A/C equipment too.
Kanawha county, a fairly large land area, has 178,000 population. I know of several smaller phone company switches in the metro area, brick buildings with no windows, 2 or 3 stories, but the downtown switch building dwarfs those facilities.
Leto
@Brachiator: All good/valid points. I’m not sure what to do here; bring in outside experts to help craft specific legislation? Look to non-profits to help coordinate this? I’m just not sure.
@The Moar You Know: pretty much this; it should be asked why they haven’t done infrastructure upgrades, but it’s the same as why rural communities don’t have high speed access: there’s no profit in it, so they won’t do it.
@NotMax: might as well chisel it on stone and throw it up in every court house :)
@L85NJGT:
So the first time I was stationed at Shaw AFB, I was part of base communications. We took care of airfield communications equipment. Included with that was all the radar equipment (separate career field from mine, the radar guys). That radar equipment was vital stuff as without it the fly boys couldn’t do their thing. And anything that stopped flying was considered beyond a mortal sin. So one day the civil engineering boys were out digging near the radar facility for some project. The spot they were digging, there was a tree nearby. As they were digging, they hit something. Seeing the tree nearby they assumed it was roots. “Keep going!” was the command. So they kept on. Unbeknownst to them, they just severed the main fiber bundle for the base radar. All the scope dopes (air traffic controllers) radar inside the facility went blank at the same time. There was a lot of traffic in the air at the time.
Back outside, feeling good about themselves for having gotten through the “roots”, the CE boys kept going. They hit another obstacle. This time, this obstacle responded by shooting a 50ft geyser of water into the air. They had just hit one of the main water lines. Work immediately stopped. Eventually water was shut off, the hole was drained, and seeing how this was near the end of the day, the civilian contractors who maintained this stuff (fiber repair work) were called out. They made time and half on that job. I think the bill to repair that 200 pair bundle was in the mid-to-high 6 figures. This incident, along with two others specifically involving us radio guys, led to the CE commander being fired, along with several other civilians being fired. But that 50ft geyser of water is still one of the funniest things I’ve ever seen.
ThresherK
All you ever wanted to know about the AT&T microwave routes…from sixty years ago.
No particular reason, just wanted to post that incredible old map. The method is, obviously, as susceptible to natural and man-made damage as the new stuff.
Doc Sardonic
@Stevie: Acually it belongs to Century Link/Lumen. They bought Qwest a few years ago for pennies on the dollar after Qwest management augered it into the ground.
Leto
@ThresherK: Seeing a number of AFB on that site, knowing what they do, and why microwave comm was so important. This is def a neat bit of history!
Bill K
In Neal Stephenson’s “Mother Earth, Mother Board” he points out that all the Internet traffic between Europe and Asia routs through a single building in Cairo. Might not still be true, but it was then.
Citizen Alan
@The Moar You Know: Then nationalize them. Which, you know, won’t happen. But it should be our battle cry.
JustRuss
@Leto: Yeah, Congress sure could use something like an Office of Technology Assessment. Which they had, until Newt killed it. In 1995, just as the internet was taking off.
Leto
@JustRuss: I def remember that. Part of the whole sale Republican plan of killing government.
@Bill K: Don’t think it’s like that anymore, but back in 2008 this did happen:
Ship’s anchor caused cut in Internet cable
I was deployed to Iraq at the time and this also affected US comms throughout the region.
John Revolta
@Leto: Gee, every time I open my electric bill there’s a thing in there telling me to check with the local authorities about where your lines are before you try digging in your yard. You’d think your engineering type guys would try doing something like that!
trollhattan
We have this ATT eyesore in the middle of downtown, across from the Convention Center.
https://goo.gl/maps/SX8aBiHTrsLQjogZ8
Back in the analogue days it housed eight stories of switchgear. Who knows what’s in there now?
“Flash Boys” gave me a whole new perspective on communications gear strategy and tactics.
J R in WV
@The Moar You Know:
CA state government needs to use eminent domain to seize their utilities and operate them for the benefit of the citizenry, if the privately owned corporations won’t do so properly.
And starting fires isn’t properly operated utility service!
If they avoid responsibility by declaring bankruptcy, the bid price for the purchase should be zero. I’m done with utilities acting like Emperor of the lands!
trollhattan
@J R in WV:
PG&E is in their second bankruptcy. #1 jerb is to keep the shareholders whole. #2 jerb is restore management bonuses.
If the state were to take control (can it happen? dunno) the first thing we’ll probably get stuck with is the decommissioning costs for Diablo Canyon. Yay.
Their biggest missteps seem to be pissing off various judges.
trollhattan
@Leto:
There’s an engineering joke I can’t properly remember but something about how very much work it is to put in fiber optics and how very easy it is for one guy on a backhoe to completely undo everything.
They take USA clearances very seriously and those are sadly never 100% accurate. Every workplan I’ve been part of includes hand-excavating around all utility lines. Surely they always follow the workplan, right?
Leto
@John Revolta: Part of the post accident investigation was: where’s your digging permit? Was it signed off by all the appropriate entities? When the CE guys turned out empty pockets, they found a few more systemic issues at play wrt training/following procedures. Again, more reasons why the CE commander was let go.
germy
Leto
@trollhattan: my former boss back in England was responsible for the communications portion of the Joint Base Pearl Harbor/Hickam merger. He spoke about the costs to install a single foot of fiber and the basic estimate was close to a $1M a foot. Even with better boring technology, it’s still just crazy expensive. I know there are national standards on how far down services are supposed to be. I think for comm it’s 18″ below ground. Water/sewage is like 36″? Even when installing in relatively “clean” areas, it’s still very expensive because of the amount of human labor involved. Such as hand excavating, as you mentioned. Think about that in populated areas with already existing infrastructure? Time is $$$, friend!
Brachiator
@germy:
If the GOP leadership had any brains or guts, they would just shut this idiot down.
At some point, even he has to pivot from “I am still president” to “Send me more money so I can run in 2024.”
fancycwabs
It’s hard to say. He could have been a QAnon-level true believer in every conspiracy theory under the sun, and that could have motivated his actions, or he might (given his reported information security background) have thought he was striking a blow against PRISM and the surveillance state.
germy
…they wouldn’t be in the GOP
Brachiator
@germy:
If the GOP leadership had any brains or guts
Ha! Too true.
Unfortunately, they don’t seem to understand how they are playing with fire. This disregard for democracy will come back and hurt them, too.
Geminid
@Leto: I supervised groundwork for a general contractor, and one time the while digging a water line excavators hit a buried power cable. Two bangs, one in the ditch, one on a nearby transformer. Later the electric company admitted there was an excessively long splice laid to the side, so no fine. But that backhoe crew had their machine loaded up and out the driveway in record time.
L85NJGT
@Geminid:
LULZ – I had a fiber coring crew go through a 300 pair. They were gone in a cloud of dust. No way they were waiting around for Ma Bell.
This is probably closer to the ATC guy who went at the punch down blocks with an axe (or was it a chainsaw?). You just can’t kook proof everything.