Apparently the fight at John Ashcoft’s bedside was over a surveillance program, codename STELLARWIND, that collected Internet metadata, according to Barton Gellman’s piece in today’s Post:
STELLARWIND was succeeded by four major lines of intelligence collection in the territorial United States, together capable of spanning the full range of modern telecommunications, according to the interviews and documents.
Foreigners, not Americans, are the NSA’s “targets,” as the law defines that term. But the programs are structured broadly enough that they touch nearly every American household in some way.
[…]Two of the four collection programs, one each for telephony and the Internet, process trillions of “metadata” records for storage and analysis in systems called MAINWAY and MARINA, respectively. Metadata includes highly revealing information about the times, places, devices and participants in electronic communication, but not its contents.[…]The other two types of collection, which operate on a much smaller scale, are aimed at content. One of them intercepts telephone calls and routes the spoken words to a system called NUCLEON.
For Internet content, the most important source collection is the PRISM project reported on June 6 by The Washington Post and the Guardian. It draws from data held by Google, Yahoo, Microsoft and other Silicon Valley giants, collectively the richest depositories of personal information in history.
The Post has a good backgrounder on metadata. According to that piece, the NSA stopped collecting Internet metadata in bulk in 2011, but previous leaks have indicated that roughly a trillion pieces of metadata were collected worldwide last year.
I had to read the STELLARWIND piece a few times to understand what, exactly, is being reported, and came to the conclusion that the vagueness was intentional. Gellman has sources, apparently other than Snowden, who are feeding him classified information, but what’s been leaked so far just confirms that these programs exist, not how much they collect.
Yesterday’s disclosures by Microsoft and Facebook show counts of requests for content, not metadata, and do not break out NSA requests. That’s why both Twitter and Google are fighting with the government to be allowed to release the number of content requests made by the NSA. The question of how much metadata they’re releasing to the NSA hasn’t even been broached.
Ryan C
Everyone should listen to this week’s Security Now podcast. Steve Gibson has a hypothesis as to how companies like Facebook and Google could be collected against without their knowledge.
Nathan
I really wish people/reporters would stop using the euphemistic term “metadata.” It’s still data. Repeating the term meta- is just buying in to the “nothing to see here, move along” framing–it’s the “coercive interrogation techniques” of 2013.
mk3872
Has King Cole come online yet to apologize to his readers & followers for linking to that Libertarian trash made-up “story” by the Paulbot at CNET yesterday?
Cole & Lurie owe everyone a big mea-culpa for subjecting us all to that utter malarkey based on a Libertarian anti-science, anti-climate science idiot @ CNET.
KCinDC
I’m glad we’re getting more information about government data collection than we were, but numbers of requests aren’t necessarily very meaningful, considering how broad some requests seem to have been. In the limit, the government makes one request that covers all communications by everyone over all time and never needs to make another request. OK, that’s not what’s happening, but an order that covers all phone metadata and is routinely renewed every few months is getting closer to that than it is to a warrant.
mk3872
@Nathan: That would be convenient, wouldn’t it?
Except it would also be wrong.
Knowing who you called and when is not at all the same as what you said.
It doesn’t really take a genius to know there is a difference.
tofubo
happy bloomsday nontheless
Emma
@Nathan: No. Metadata is technically separate from data. Currently I am involved in setting up metadata files on thousands of articles on legal subjects. I collect author, title, publication, and subject data. I don’t read the articles and wouldn’t understand most of them if I did. This will then be able to be sorted by researchers into subject files. The experts, I am sure, will be able to detect patterns I can’t see.
When the government collects phone metadata it’s collecting the equivalent of your phone bill. If it detects a suspicious pattern, it needs a warrant to read the content. IF they are bypassing that step they are doing so illegally.
(edited for clarity)
D58826
Funny (sad funny not haha funny) that the GOP’ers screaming about the end of the republic with regard to what still looks like the inept handling of the 503 application by the IRS are more than happy to let the NSA intercept and read every piece of electronic communication in America.
Brother Machine Gun of Desirable Mindfulness (fka AWS)
@mk3872: maybe you should e-mail him.
cleek
as a good liberal, should i be upset about this ?
Professor
@Emma: Thank you very much. At present, a lot of people (libertarians, some emoprogs and rightwing nut jobs ) are misrepresenting Mueller’s congress disposition!
Ultraviolet Thunder
@Emma:
Digital photographs have metadata as well. If it’s from a cell phone it will have date/time as well as location to the resolution of the GPS system that’s enabled. There’s probably a lot more in there to identify the origin of the photo.
I use cell phone photo metadata to prove when/where I took a shot for business purposes. But all of that is floating around for other people to use if you don’t strip it out.
Emma
@cleek: I will take your snark seriously: I don’t know if liberal comes into it, but as a citizen, you should, but not from THEGOVERNMENTISCOMINGTOGETUSALL mode.
There are two big problems here. One is that as a society, we haven’t yet come to an agreement as to what “privacy” is in an electronic world. The younger generations routinely broadcast what I would consider inappropriate personal matter in public social media sites. Can that we considered “private” any longer? If so, how can it be protected? And are we talking about the government or about all the routine business collecting and redirecting for merchandising purposes? After all Google and all its relatives, plus AdSense and all its relatives are likely to have more information about you than the government has.
The second problem is how much “freedom” are we willing to tolerate. If we decide to tell the government to back off surveillance unless tailored to an individual suspect we are going to have to accept a hell of a lot more uncertainty in life. From every poll I’ve seen, Americans will rather be safe than free. How do we change that? (other than eviscerating the media with their “war and scandal” drums?)
Oh and one more. How do we make sure Congress is doing its job? We have seen this Congress pretty much in “if the President wants it, we’re not going to do it, and we’re going to effectively stymie him” mode since Obama tried to close Gitmo. Again, how do we change that? It means changing the electorate and it really took the Conservatives 30 years of steady propaganda to raise crazy to this level.
RSA
Yes, it gives some nice illustrations of the concepts.
@Emma: I agree. Another way to put it is that meta data is information about some other information, or content. But it’s to some extent a matter of perspective. A library’s card catalog is a traditional example of metadata (e.g., a book might be put in a given category, even if the book doesn’t literally mention that category). But for a Web silte like World Cat, which organizes information about where to find books in libraries over the entire globe, catalog information is content. So Nathan has a point in saying metadata is data. (Also, it could be seen as another example of computer jargon entering the mainstream, with not all news journalists understanding what the term actually means.)
Emma
@Ultraviolet Thunder: Exactly. All my cameras do that automatically unless I actually disable the function.
Starfish
@Nathan: I have read your meta-data, and it was boring. Actually, I worked for a place that had reason to collect meta data. It collected information like phone numbers and time and duration of call to see that calls were going through properly. If calls were being dropped, it was possible that there was a bug in the system that needed to be fixed.
Poopyman
@Nathan: Metadata is data about data, and as such is more descriptive than just using “data”.
Emma
@RSA: What WorldCat does is give you information about where something is and, depending on the record’s creator, a good idea of what it is about. So it’s a aggregation of catalogs that facilitates locating an item. In fact, I would consider it the (nonsensitive) equivalent of what PRISM sounds like. I can check a title, its editions, its versions, and where I can locate them. It still tells me what the title is about only in general terms.
So if I’m looking for a recording of the (1970s) version of Much Ado About Nothing with Derek Jacobi, it will tell me if someone has it and if I can borrow it. So you can get data but it’s not content.
Excuse me, I’ve got to go check WorldCat….
max
@Emma: When the government collects phone metadata it’s collecting the equivalent of your phone bill. If it detects a suspicious pattern, it needs a warrant to read the content. IF they are bypassing that step they are doing so illegally.
The problem being is that it makes it very easy to a) abuse (for example an NSA contractor deciding to find out who his wife has been talking to) and b) they appear to be bypassing that step, which comes as no surprise.
If we decide to tell the government to back off surveillance unless tailored to an individual suspect we are going to have to accept a hell of a lot more uncertainty in life.
That is, about as much uncertainty as existed in the year 2000, which I, for one, was totally OK with. Thomas L. Friedman is totally not OK with that as his Depends are soaked at the very idea.
It means changing the electorate and it really took the Conservatives 30 years of steady propaganda to raise crazy to this level.
Since I’ve been in vehement opposition to this whole shebang since they year 2001, more or less, I think I can say that I am not the kind to give up. And thus, if it takes patience, so be it.
Mastermix: Gellman has sources, apparently other than Snowden, who are feeding him classified information, but what’s been leaked so far just confirms that these programs exist, not how much they collect.
Well, yeah. Leaking is FINE and TOTALLY OK and JUST DANDY as long as they people leaking are people who pro-NSA/pro-surveillance and thus making everything sound perfectly OK, 4th Amendment be damned. That’s not TREASON, because Jesus TOTALLY was all like, ‘Make sure you get your contractors some extra cash so they beat the infidels and get into Heaven.’ Because A NOUN! A VERB! AND 9/11!
max
[‘They’ve been running the same play for 12 years so it ain’t like I ain’t used to this.’]
RSA
@Emma:
Sure. I was assuming that WorldCat maintains its own database of catalogs, and if that’s the case, then from the perspective of their system developers, they’re working with data (which other systems consider metadata).
Nathan
@mk3872: Except that’s not how the “meta” data is being used… it’s a guilt-by-association. You’re just talking about a finer grain set of data “well they know I had dinner, but they don’t know what I ate, so it’s ok.” Knowing that you ate dinner and when… IS STILL DATA. Surely you can recognize that.
So, if a suspicious car parked across the street from your house with two guys who seem to write down every time you come and go from your house and who you’re with… that wouldn’t bother you–you know, since they’re not actually in your house? And wouldn’t your next thought be “how soon before these guys break into my house when I’m not home?”
Sister Rail Gun of Warm Humanitarianism
@max:
So you missed the analyst who talked about people getting fired for checking up on the ex? And the keyloggers installed on the analysts’ computers to catch that sort of thing?
Nathan
@Emma:
Uh that’s still data. Your argument about not understanding the content is like saying “I can’t hit a curveball, so the batting averages I compile are not useful/meaningful”
Sister Rail Gun of Warm Humanitarianism
@Nathan:
You’re not on Facebook, are you?
Not too long ago, Facebook finally gave in and let its users block other people from “checking them in” places. Apps that post real time data on your jogging routes are popular. That data is already public for am awful lot of people.
Emma
@RSA: WorldCat doesn’t maintain a database of catalogs. The first “WorldCat” was created in 1967. It was originally intended as a cataloging tool and it was set up by the Ohio College Library Center to connect Ohio libraries. After a while, it evolved into the Online Computer Library Center, Inc.Participating libraries contribute catalog records than can be borrowed by other members. Most libraries in the US and some abroad are members. Adding holdings information became common after libraries started using it as a interlibrary loan tool.
WorldCat, as we know it, is the public face of that database. As libraries include more records, including records that point to open access full-text materials on the Internet, it continues to evolve — like most things in libraries.
RSA
@Emma: Thanks for the information. I used a bad example, then.
Suffern ACE is a Basset Hound
@cleek: kind of. It should be unsettling. Doesn’t mean you need to go full on paranoia meltdown about “the government this and the government that.” The CIA, FBI, NSA, DIA and whatnot are not exactly instiutions that serve your goals and haven’t had long histories of, say, advancing the cause of every day folk for social and economic justice. I don’t mind glancing over at them from time to time making them explain themselves.
Emma
@Nathan: No, it isn’t. It’s the equivalent of “I can collect information about legal materials but I don’t bother with the content.” Look. Take your phone bill. I can look at your phone bill and see that you call a certain number every Friday. Yes, if I cared to do into it in depth I could find out who you were and who you called and all that. But why?
There isn’t enough manpower in the whole federal government to do that for every citizen of the US. Can it be misused? Hell yes. But so can meatspace repositories. What do you think debt collectors do? Or lawyers in divorce disputes? Private detectives?
Emma
@RSA: No problem. Library arcana :D
Nathan
@RSA: RE: computer jargon… exactly. I’m a software engineer and we have always used the term “metadata” as the information that literally describes the structure of tables/data/format–so I do see the use of the term to be a perversion of the original and a perversion intended to obscure.
I think where I object to the metadata term w/r/t “well they don’t know what was in the call” is that the call or email content is not properly “data” but “information” at that point. Yes and probably the popularization of calling your photos and videos of your kid’s Bat Mitzvah “data” has not helped.
NickT
@Nathan:
No, it isn’t guilt by association. It’s pattern recognition – does this person have a pattern of visiting suspicious websites or calling known adherents of terrorist groups/terrorist sympathizers. If they do, it’s worth taking a closer look. This is not the same thing as declaring them guilty.
Emma
@max: I am going to address those parts directed to me.
Can it be abused? Hell, yes. Which is why I keep talking about tightening controls on FISA.
You and I are willing to live with uncertainty. The majority of the American public doesn’t and educating them to the realities of modern life will be damn difficult, considering how many years the conservatives and the media have been beating “the browns (muslims, etc. etc. pick your group of choice) are coming to get you” drums.
I am actually with you on the substance, you know. I want the Patriot Act gone. I want to know, even at several years remove, how FISA made its decisions. Transparency, transparency, transparency.
I am not yet, however, ready to run around tearing my hair out about the police state, especially since we have conflicting stories coming from everywhere.
monkeyfister
Rumsfeld’s and DARPA’s love child, “Total Information Awareness” is alive and well and never ever went away– just like Rumsfeld said while laughing at America. He told us straight-up that they’d change the name and ditch the logo, but it would continue, and now in TOTAL secrecy.
It did… and here we are.
RSA
@Nathan:
That’s the way I automatically think of metadata, too. (I’m a computer scientist, though not in software engineering or databases.) So you already knew everything I was trying to say by using a bad example. :-)
And I’m also annoyed when technical terms are used in non-technical settings to hide more than illuminate.
Tripod
@cleek:
If by good liberal, you mean occasionally Democratic voting Libertarian? Then yes, it seems you should be very worried.
Nathan
@Emma:
Yes, there is just a slight difference between cataloging/tagging freely available information and cataloging/tagging not-so-freely-available information. This is exactly the point… what gives ANYBODY the right to look at my phone bill and know who I called on Friday?
And if you think it’s so great, send “ME” your phone bill and your internet search history… you know, just in case “I” am interested… “I” probably won’t be, but keep mouthing off and “I” just might, Don’t worry “I” have a lot of other people to intimidate first, so you’re probably safe for awhile.*
*Now just substitute “NSA” or “government” for the personal pronouns. You wouldn’t give me that information and I’m a perfectly harmless stranger…why would you freely surrender it to a government entity that has the entire power of state behind it?
NickT
@Nathan:
There’s a fairly substantial difference between handing over your data to some random person on the internet and a government agency looking at your metadata. And yes, metadata is a perfectly legitimate use of the term here. Language evolves, as it always has, and the fact that back in the day only computer geeks used metadata (or, more accurately, structural metadata) isn’t really relevant here.
Cacti
@mk3872:
The Supreme Court decided that particular issue in 1979.
Nathan
@NickT:
cf. Driving while black.
NickT
@Nathan:
Not the same thing at all – which is why you couldn’t even make a pretence of an argument here.
Cacti
@Nathan:
Everyone knows that white internet users are the real persecuted group in this country.
Suffern ACE is a Basset Hound
@Nathan: I think what is going on here is that different agencies in the government are allowed to see different things and have to obtain different approvals to obtain those things. So it isn’t me giving you everything. It’s me giving you my search history but not my name and giving Emma my phone metadata but not my phone number and giving cleek my phone number and email address and if he wants more he needs a warrant and if he wants to get my search history he needs to come to you and you need to decide to give it to him otherwise he needs some kind of an order that gives him permission to try to get it himself.
NickT
@Cacti:
White libertarian internet users, that is. Unless the legendary persecuted white male Christians want to apply for their usual victim status.
NickT
@Suffern ACE is a Basset Hound:
Right, which is exactly why Nathan is so keen to claim that all forms of data lead to instant guilt by association by fudging the difference between data and metadata. Once you realize that there is a difference, the NSA looks just a bit less like some sort of fantasy panopticon aimed at targeting people who buy spinach on Sundays.
Cacti
@Nathan:
That question was asked and answered in Smith v. Maryland (1979). You should read it sometime.
Suffern ACE is a Basset Hound
@NickT: freedom is red tape!
NickT
@Suffern ACE is a Basset Hound:
Freedom is just another word for no metadata left to lose!
Emma
@Nathan: You are not entitled to my metadata. You’re not entitled to anything. You are just another internet asshole like the rest of us. The government is CHARGED with protecting the country and they have developed these tools and some have even been declared constitutional by the Supreme Court.
Now, it might be that as a society we will decide the government is not entitled to any of it either and we will declare FISA, the NSA, the CIA and all other such agencies unconstitutional. Or we will do the hard work and really decide how to balance security and freedom, create appropriate rules that will be appropriately applied with appropriate supervision.
Let us find a Congress that will do that.
Suffern ACE is a Basset Hound
@NickT: it is a valid question as to whether anyone can see it all (who also can do something to you) without running through any bureaucratic checks. So even if the NSA could, it’s not like the supposed super analyst can also hit control-alt-S and send a guided drone to your house.
Liberty60
@Emma:
Right, and part of “doing the hard work” as citizens is having the sorts of arguments we are having right now.
In order to have these arguments, we kind of need to know what is going on. Hard to do when virtually everything that the NSA does is shrouded in secrecy, plus they get caught lying from time to time. Add to this the history of Congress legalizing even the documented violations of lawthat do occur, and I think it is astounding that anyone could grant the government the presumption of innocence.
Needless paranoia isn’t helpful either, but I think its entirely reasonable for us to be skeptical of what they are up to, and extremely concerned.
Howard Beale IV
@Ryan C: Which leads to the next question:
Why isn’t the traffic between two backbone peer’s not encrypted at the point of the tap?
bemused senior
In the discussions in this and other articles about collection of
data from the routers, people need to realize that large internet providers use their own fiber to connect their data centers, and also optimize things to avoid shuttling large amounts of data between data centers. So the routers serving internal traffic are largely within the providers own premises and not in ISPs. Of course the fiber itself is physically available for tapping, assuming data has to be shuttled, but a large fraction is not.
Edge traffic, e.g. accessing the services from your home, is different. However mail, at least the gmail many of us use, is sent to the Google mail servers using transport encryption (https). While this isn’t a magic bullet to ensure privacy in transit, it does make it very computationally costly to target such data in bulk, requiring special attacks for data coming from different user platforms, and I for one doubt that such mass targeting/collection is done. Since the mail metadata is hidden within the encrypted transit stream, it is not available to router based data collection either.
Posting on forums and commenting on stuff (“social media”) doesn’t transit using https, so it is available to snooping at routers in the ISPs.
Emma
@Liberty60: Oh, I agree. If you go back over my postings in the last few days you’ll find that I am an advocate for a number of things, including more transparency. We need, pardon the expression, data.
Howard Beale IV
@bemused senior: So they’re no coincidence whatsoever that PRISM could be/is splitting the optical traffic between the Tier 1 backbone and their peer gateways?
Nathan
@NickT:
What is a “suspicious website” then? Who determines what a suspicious website is? If I visit a suspicious website 1x,am I in trouble? If I visit 0x? If I visit it but don’t leave comments? Are these suspicious websites well-known, published “don’t go there?” so I can be a good little citizen and not go there? Do I get 3 warnings?
Or is it like pr0n, Justice Potter, you know it when you see it?
So racial profiling invalid, data profiling valid? I don’t get it. How can you tell the difference between a student researching a term paper on Neo-Nazism and a dangerous radical looking to join up? Or do you just have to keep gathering more and more data to be sure? Medical history? Credit-card statements? Camera surveillance?
bemused senior
@Howard Beale IV: Not PRISM — do you mean stellar wind? Data flowing over the backbone is data that is routed between providers (such as from your ISP to the big web mail providers) or between data centers that use the internet backbone for internal transmission. This is why I distinguished between edge traffic vs. internal traffic, and encrypted traffic to providers vs unencrypted.
Emma
@Nathan: can you tell the difference between a student researching a term paper on Neo-Nazism and a dangerous radical looking to join up?.
Look. Most of those things are irrelevant. In so far as I know, it isn’t illegal to join groups. I am sure millions of people Google/search for Al-Qaeda, terrorism, all those trigger words, everyday. But unless and if it is part of a pattern, it is below the radar. Truth is, most American lives are patterns of “yawn,” as far as criminal behavior goes. In order for you, for example, to set off bells you’d have to do at least several things that have been proven to be markers in the search for terrorists. A high-school kid looking at Wikipedia ain’t it.
Nathan
@Cacti:
Yes 35-year old court decisions that overturned decisions just 10 years previous are complete proof that the law and its interpretation are set in stone and will never ever change.
I missed the part covering my internet search/web history and email correspondence. What did Smith have to say about that?
It seems to me that Smith narrowly covers a means of collecting data [in this case, phone records] being constitutional or not…whether NSA comes in the front door or the back door is quite cosmetic/incidental to the question of access/use of the data in the first place.
bemused senior
@bemused senior: Should have said MARINA.
PRISM is the system that deals with data obtained via requests by user ID for user generated content stored at the big internet providers.
Here is the best discussion I’ve seen about PRISM, and it comports with what I know as a former employee of one of the providers.
http://m.washingtonpost.com/blogs/wonkblog/wp/2013/06/12/heres-everything-we-know-about-prism-to-date/
FlipYrWhig
@Nathan: This is a bit like saying if you’re willing to be patted-down by TSA at the airport then you must be willing to be groped by random strangers. Officers of the law are authorized to be more personal and intrusive than Joe Schmoe, and that’s been true for millennia. That’s not to say that everything an officer of the law does is totally cool, just to point out that your analogy is lousy.
? Martin
@bemused senior:
Exactly. They can collect the originating and destination IP, the port numbers (which strongly hint to the contents: 25 is email, 80 is web, etc.), but the data is often just a subset of what’s being communicated. The packets need to be reassembled. And what’s important on the data encryption is that it’s endpoint encrypted – between my computer and the machine at the other end. There’s also router encryption (which everyone should have turned on their home router, right?) which provides encryption on top of the endpoint encryption (so that HTTPS session is encrypted twice, once by your browser, once by your WiFi card, then the WiFi encryption is decrypted leaving just the browser encryption and sent down the wire, possibly re-encrypted by routers along the way). A law in 1994 required routers to provide a back door to law enforcement so they can bypass the hardware encryption if they have physical access to the router. But the browser/mail client/whathaveyou based encryption is quite secure.
The trick with the email encryption (TLS) is that you can require it be encrypted at your end, but where the other end decrypts it, and how securely they route it after that happens is out of your control. It’s possible your email will be grabbed by a border filtering machine (looking for spam, attacks, and so on) decrypted (to do the above items) before being sent to another machine, and then being sent to your delivery machine (the one that actually stores the email). Those last legs can be entirely unsecure. Normally, those machines are all in the same room, possibly in the same rack, but large organizations will have to spread them out. I don’t really trust ISPs to not fuck that up – to not do the filtering in one location and then route to regional delivery points in different buildings and not re-encrypted the message when it leaves the building, travels via god-knows-what-path, before arriving at the delivery point. This is why organizations that need secure email, don’t let ANYONE else touch it. The protocol does not guarantee the encryption will remain the whole path. Really secure organizations will put a layer of encryption above TLS in the client before it’s sent (something like 3DES) and again at the other end. This doesn’t encrypt the headers, just the content. This helps ameliorate the above problem, but not entirely.
The NSA can break it, but they need to be quite determined to do so. It’s going to take them days/weeks to decrypt one message.
Regarding the social media stuff, until last year, Facebook didn’t encrypt your login credentials. That was sent in the clear, so it was trivially easy to intercept that (particularly if you set up a fake public hotspot and ran a packet filter) and then just access your account. Armed with a login and password that people type constantly, you could pretty easily run a script to hit all of the other popular sites from amazon to twitter to see if the same login/password would work. At each step, reset the password to hijack the account until the owner got around to resetting it. That was easy enough for a teenager to do, let alone the NSA.
@Howard Beale IV: I doubt that the NSA is collecting metadata en-masse and doing anything with it. It’s a seriously massive volume of stuff, and tells you almost nothing. Now, they might be filtering specific traffic from/to suspicious destinations, but grabbing everything would be beyond pointless. Every person watching a movie off of Netflix generates roughly a million packets. That’s your first million pieces of metadata, and that’s duplicated at each hop. There’s no choke point on the internet so they’d need to copy each packet repeatedly at each point where it hits a tier 1 and probably a tier 2 as well. For communications not between sites that sit on a tier 1 (like Netflix does) then multiply it by 3 or more.
Gangnam Style is about 6MB. It’s been viewed 1.6B times. That’s 6.4 trillion (with a T) packets of nothing but one Psy video just in the last year. Nobody is going to bother capturing the metadata. Some of it, sure, but it’ll be fairly selective.
Jasmine Bleach
@Emma:
“A hell of a lot more uncertainty in life”? Umm. No. This is the false impression media and government are trying to push onto people. Thanks for helping to push it. Even with all this “security”, the Boston bombings happened and nobody knew about it beforehand.
How common are terrorist acts? Not common at all. You probably have a better chance at winning the grand prize in the lottery than being killed by terrorists in your life.
Definitely tailor it to probable cause and individuals. That’s how it should be.
bemused senior
@? Martin: It seems unlikely that the router interception is operating at the packet level. I would think netflow is a more likely “smallest unit”. http://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html
Even that generates enormous amounts of data. Have worked on a system for a large hosting provider that did its charging based on netflows, and preserved the raw data for a (shortish) period of time. The operational issues with consolidating and storing this were enormous, much more severe than the problems with processing it. We were doing this system because the older custom system wasn’t processing 24 hours of data in 24 hours, and commercial systems weren’t able to keep up at all. And all this data was used for was aggregating the amount of traffic to the set of IP addresses representing the big companies being hosted! Yes this was 10 years ago, and systems and storage have improved, but in our case the data was only important as to the endpoint at the hosting provider. If you are looking at individual origin points this is truly an enormous scale )
Emma
@Jasmine Bleach: No. I am not pushing it. Please become familiar with what I have written before starting the nasty punches.
It will be more uncertain because it will be seen as more uncertain. It will also become more politically volatile UNLESS the population in general accepts that there might be a chance of another attack and is willing to live with it. As much as it pains me to say it, the majority of the American people would do anything to avoid another 9/11 type attack. They have been sold on the security state. We have to find a way to unsell them because any of this becomes more than an academic discussion.
What “should be” and $5 will buy you an iced latte at Starbucks.
Suffern ACE is a Basset Hound
@Jasmine Bleach: like it or not, 9/11 was a very serious catastrophe and a failure of the “security state.” And I don’t think being concerned about preventing another one is the same as asking the government to prevent a thunderstorm. Perhaps we should ask the Terrorists to please stop communicating with each other and try to be more detectable.
Iraq was the country losing its collective shit in response to 9/11. The shoe removal also. But I’m not certain that the data collection is. Sorry if you feel mildly inconvenienced.
Howard Beale IV
@? Martin: BusinessWeek, May 23, 2013:
Anybody have any idea what the total bandwidth is of the Tier 1 backbones?
bemused senior
Try this:
http://www.telegeography.com/page_attachments/products/website/research-services/global-internet-geography/0003/1871/GIG_Executive_Summary.pdf
ETA:
http://en.wikipedia.org/wiki/List_of_Internet_exchange_points_by_size
Ryan C
@Howard Beale IV: @Howard Beale IV: That’s how the Internet works now. I wish it wadnt. But I’m glad someone else here is paying attention to the actual technologies involved. Thank you! It’s refreshing to hear about issues instead of the tribalism and petty political sniping.
Ryan C
@Emma: Since you’re in the know, what is “it”? I’d like to know when the constitution gets shredded. It’s amazing that we have a commenter here with total knowledge! Do tell!
Chet
@Nathan: Well, it’s data about data, or meta-data. It’s being called metadata because that’s what it is.