I don't see how it is possible to record all communications, 24/7. 365/year on all Americans.

[IsItJustMe]

This has to be tin foil. The technology just don't exist. This would require huge warehouses of storage and retrieval systems, with a huge number of super computers (like the Cray) hooked together in extremely complicated series of networks.

From my understanding, many government computer systems are antiquated, at best. I think this is being put out there for the fear effect that it will have on anybody that might get an idea. But, I for one, just don't buy it.

[grannie4peace]

1. ?what are you talking about?

[IsItJustMe]

4. All American domestic communication being saved.

[MineralMan]

2. Don't underestimate the NSA.

I worked there while in the USAF, way back in 1968-69. The capabilities, even then, were astounding. I doubt they have diminished in the passing years.

Their computers cannot be equated to those of other agencies, I guarantee.

[arcadian]

8. I don't doubt they have the capability to pull and store that information.

But they damn sure don't have the human power to analyze it.

[WeDidIt]

14. They don't use "human power"

They use stuff like http://www.spss.com/predictive_text_analytics/">This

[arcadian]

53. At some point in the analysis process a human has to be involved.

To enforce "real world" situations.

[WeDidIt]

54. That's after they filter out the noise n/t

[Uncle Joe]

57. Due to technological glitch this post has been self-deleted. n/t

[Uncle Joe]

58. I would also imagine humans would need to program what is to be filtered

out or what constitutes noise.

I'm not a techno geek, but it seems to me corruption could take place there during the shifting as well as final analysis

[MineralMan]

60. Human power? It is to laugh.

Everything is analyzed using very, very good voice and text analysis software, filtered, analyzed, and put in appropriate priority levels. Then, humans have a look at the tiny percentage that meets whatever interest levels are set.

They were doing that in 1968. How much better they must be at it now, forty years later. The NSA has driven a lot of advances in computer tech. They were miles ahead when I was there. They're still, no doubt, miles ahead of what is commercially available.

As an example, I have no doubt they can resolve an IP address to which computer in your house or office was involved in a connection. My computer's name is George. It's one of three on the network. I wouldn't doubt that they can resolve to that level.

Using an anonymizer? Silliness. If they're not running the anonymizers, they can access the origination IP addresses of any user. I imagine the anonymizer users get a higher priority than the typical nerd.

Virtually 99.9999% of everything they track gets dumped automatically. The other .0001% gets looked at, at some point.

Don't be confused.

[annabanana]

3. the DoD has the VERY best & most up to date equipment

- -much of it still unavailable to anyone else - -

Who knows the state of nano-technology these days...

[IsItJustMe]

6. If the data storage technology exists, then they have not let commercial interests in on it.

Because from what I see on the market, that capability or capacity is just not out there.

[danalytical]

7. I agree

There is more communications in one day in the US then all the storage capabilities on the planet. They must be saving/recording only when targeted words are written or spoken. Or all communications of only targeted people.

[IsItJustMe]

21. If this were true, much of the communication data would have to be stored off line. If they

wanted to keep it all online, I would say that they would have to add several Super computers to their network every day. That is just not feasable or practicle.

Given the fact that much of this data would have to be stored off line, if a search were to be made, it would take days or even weeks for the results to come back.

The resources that it would take to maintain such a network are unimaginable.

[hughee99]

55. I'm not sure about that...

If you can store 200+ TB in the space the size of a fridge, the capacity is out there. I think the ability to collect all this information is more difficult to believe than the ability to store it.

[meegbear]

5. You may be correct about the goverment ones ...

and that's why companies like AT&T are/were doing it for them.

http://arstechnica.com/news.ars/post/20071111-ex-att-employee-nsa-snooping-internet-traffic-too.html

[lunatica]

9. AT&T and other telephone and telecommunications companies have been

working with the Bush administration to spy on us for years now. And from what I've heard they do have the equipment to spy on everyone. There's no reason each company wouldn't have massive computers to spy. And the software they use checks for certain words and connects patterns, so there's no need for a person to actually sit and listen to the telephone conversations.

Bush recently (within this year maybe) passed a law protecting these companies from being prosecuted because they were asked to spy on us by the President and the government

[seabeyond]

10. years ago...... 3,4 we had the info. i dont remember it all. do research. and yes

they are.

it grabs hold of certain words used in conversation. but the info is out there for all to "see" so they can "believe"

[FreakinDJ]

11. Largest perveyor of CRAY SUPER COMPUTER

They have Cray Super Computer Complexes like Goggle has Server Complexes all over the world

[NeedleCast]

17. And?

Even if they did, and they don't, Cray type computers are mostly used in scientific calculation and simpulation development.

Cray makes some very advanced processors but we're not talking about processing power, we're talking about storage capacity.

[FreakinDJ]

19. Check you facts CIA/NSA largest perveyor of CRAY

sorry don't get where you think that is not a fact

got a link to support your theory

100s if not 1000s of these
http://www.cray.com/Home.aspx

Running software that does this
http://fas.org/irp/program/process/echelon.htm

It has been well documented for a long time

[NeedleCast]

28. Even with 1000s of terrabyte servers (which those links are not)

you couldn't RECORD and STORE even a small fraction of communication. Cray's are very powerful computer PROCESSORS but that doesn't mean they have an abnormal amount of data storage capability.

I don't think that if every storage device on the planet were suddenly put under NSA control that they could store a single days worth of global communications. Understand that several BILLION E-mail messsages are sent per day (95% of which are spam), and that E-mail comprises only a fraction of global communication. Doesn't matter if you've got ever Cray on the face of the earth crunching E-mail, you'd still only be able to sift through a fraction of it and then you have to have actual manpower resources dedicated to reading what the softwart pings on.

And that's what the OP is suggesting. NSA may have the ability to crunch all the data they record, but it's impossible to STORE all that data, thus, they use targeted operations.

Please don't read this as support for NSA wire-tapping, it's not. At the same time, as a person who worked in defense intelligence for about 10 years, I know a little about this stuff.

[FreakinDJ]

30. My understanding is they only store Targited and Alerted

communications

Which yes they have the resourses to do.

These are huge multiple data farms positioned around the world

[NeedleCast]

42. Yeah, we're on the same page I think

I'm sure there are huge, multiple data and storage farms positioned around the world, but even then, the amount of voice and data communications that takes place in a single day is staggering. You'd have to have targted communications and sophisticated software to sift through it, and I'm sure the NSA does.

I'm just agreeing with the OP in that they don't have anything that even approaches the capacity to intercept and store it all.

[IsItJustMe]

33. Thank you for understanding and intelligence on this matter. You get it.

[IsItJustMe]

31. It would take a super computer processor, only avail abe in computers like the Cray, to store,

catalog, and retrieve the vast amounts of communication data. Extensive memory algorithms and memory pages would have to be accessed and utilized. Only a super processor could handle such a task.

[Statistical]

34. More like IBM or HP

Cray is old school. Their claim to fame was single machine power and era that is long since faded.

Sure a Cray single machine computer may be 1000x faster than average PC but companies like IBM & HP figured out that it doesn't matter if you network together 100,000+ PCs.

Top500 supercomputers
IBM Share: 37%
HP Share: 42%
Cray Share: 4%

http://www.top500.org/stats/list/32/vendors

[IsItJustMe]

35. Well, I am old school and I guess that is why I have Cray stuck in my brain. I see your point.

[NeedleCast]

43. I thought Cray had gone out of business years ago

I'm surprised they didn't, but I see from reading through some of their web page (which is quite interesting if you're into IT stuffs) that most of their computers now seem designed for scientific number crunching in a stand alone or limited network.

Processor wise, you'd probably be better off with a large network of standard, over the counter, common computers all crunching the same data. I know back in the late 90s and early 00s when I was working with satellite imagery analysis we did have some main-frame computers (Silicon Graphics machines) which stacked multiple processors on a single board (eight, in our case).

[Statistical]

46. Yeah that is how all super computers work.

The really are not "computers" in the traditional sense of the word more like "computer networks".

They are custom built computer boards with 4 or 8 processors per board.
Stack the boards in a cabinet (32 boards per cabinet) so you have 100 to 300 or so processor per cabinet.
Then network the cabinets. The largest supercomputers are a network of 400-600 cabinets each containing 100 to 300 processors.

Cray is still around they build smaller monolith machines but are moving into parallel computing.
The market is completely moving to large computer networks (currently they hold the #2 spot built for DOE Oakridge National Lab).

The Top 500 list I was looking at was old. Cray has better showing in the most recent list. HP & IBM are still the giants though.
http://www.top500.org/list/2008/11/100

The problem for Cray is that they are no longer "special" they use the same parts as everyone else now. Their designs while good are no longer exceptional. More and more supercomputing is becoming a software and network problem.

The top super computer uses 129,600 cores. There is no hardware reason why we couldn't build one with 200,000 or 500,000 or 1,000,000 cores. The problem becomes developing software and networks to effectively use that power.

Both IBM top design and Cray top design have max usable performance that is about 75% of theoretical performance. IBM does it with 1/3 the power compared to Cray. Power consumption for networks this large becomes a huge portion of operating budget.

The performance is getting "crazy" though.
Cray Y-MP broke the Gigaflop barrier in 1988 (billion calculations per second) in 1988.
ASCI Red broke the Teraflop barrier in 1996 (trillion calculations per second) in 1996.
Both Roadrunner & Jaguar broke the Petaflop barrier (thousand trillion calculations per second) in 2008.

So roughly every decade supercomputing performance is increasing by a magnitude of 1000x
Every 20 years performance has increased by a magnitude of a million.
Calculations that would take Y-MP a WEEK to complete can be processed by Roadrunner in <1 second. Insane.

[NeedleCast]

48. Very interesting, thanks for posting that

[NeedleCast]

12. Even if they could, how would they listen to it all?

I'm sure it's targeted, but it's the methods of targeting that are questionable.

The technology to record and store it all doesn't exist yet. Even with giant terrabyte storage farms, you couldn't record a fraction of one percent. My organization moves 100+ gigs over our servers daily...and we're talking about a non-profit that employes about 45 people. That doesn't count phone conversation, only E-mail and other web based data transfer.

Many government computer systems are older, but that doesn't usually apply to the defense industry. Still, even with the very best, it's hard to imagine even a tiny percentage of data being recorded and even less of it actually being listened to or analyzed.

That still doesn't make unwarrent wire taps a good thing.

[dorkulon]

13. They don't save it all; they filter it.

Based on key words or phrases, and also specific contacts. Of course they can't save everything, nor could they possibly review everything even if they could. The point is that they have access to everything.

[CreekDog]

15. why not?

they did it in Batman.

[FreakinDJ]

16. Here is how they do it

100s of these
http://www.cray.com/Home.aspx

Running software that does this
http://fas.org/irp/program/process/echelon.htm

[Statistical]

32. Cray? What did I go back in time to 1980s?

By late 1990s the idea of dedicated high performance machines (of which Cray was one of the best) had faded.

Modern super computers are instead a massive array of parallel processors. They use similar low performance processors that you have in your home PC but instead of having 1 or 2 they have thousands.

In the 1980s & 1990s Cray lead the way with single machine high performance computing. They hit a brick wall in late 90s though. Parallel computer using hundreds then thousands of general purpose CPU caught up to single computer designs, then passed them, then blew them away by a magnitude.

Designs by Cray today are nothing special. They are similar to all other super computers (a network of thousands of processors). The era of single high performance machine has passed.

IBM BlueGene/L has 131,000 processors spead across thousands of individual computer boards and hundreds of cabinets connected by miles and miles of networking.

BlueGene/L is 9x more powerful than the largest Cray design (a parallel network computer).
It is a staggering 200x faster than the most powerful single machine Cray design.

IBM rumored next supercomputer will break 500 Teraflops (about 400x more powerful than most powerful single machine design).

IBM is the clear leader in parallel high performance network computing. Over half of top 100 super computers are IBM designs.
http://www.top500.org/list/2006/06/100

[TahitiNut]

56. Most folks don't have the foggiest about massively parallel processing ...

... or how topological indexing works or how petrabytes of data are stored and accessed. Even a long-time MIS/IT nerd like me has had little contact with it.

[L. Coyote]

18. And I can't see Pluto. So, there is no Pluto!

Try taking a computer class.

[IsItJustMe]

26. I have, many.

[Brazenly Liberal]

20. spying on everyone doesn't mean spying on everyone and saving everything doesn't mean...

...saving everything.

There are a couple of ways to do this:

1. Filters. Have electronic sniffers watching for code word/phrases that are of interest and only save those communications. You are effectively spying on everyone, but only saving some communications from some people.

2 Random sampling. Think of companies who do random drug testing on employees. Usually, they only check some of the employees each time. But all employees carry the same risk of being tested each time. Similarly, as long as you are in the pool of people subject to random sampling, you are effectively being spied upon.

[MadHound]

22. Let's see, suck in corporations like ATT to do your bidding

Add in acres and acres of computer storage and analysis in an underground facility outside of Langley VA, and yes, it is quite possible, no tin foil needed.

[Statistical]

23. Where did you get the idea (conspiracy) that govt saves EVERYTHING.

That isn't how signal intercept works. 99.99999999999999999% of communications are worthless from an intelligence perspetive.

How Echelon and other programs work is they scan in realtime a significant portion of electronic, fax, and voice communication. In the 80s & 90s a large % of trans-atlantic voice traffic was via satallite and sat intercepts are painfully easy. They just pull it out of the air (well tecnically space).

The goal isn't to archive everything. I have never even heard the most tin foiled conspiracy theorist propose that.

The goal is signal INTERCEPT. Echelon tags and saves specific communications based on keywords, prhases, country of origin, people on watch lists, etc. A tiny fraction of world communication is every saved.

Even among those tagged by software like ECHELON most is garbage. Human Intelligence officers go through that to essentially find the needle in the haystack.

So you are right it is impossible but nobody except nutjobs ever though in the first place the govt STORES everything. The concern is more how much they INTERCEPT.

[IsItJustMe]

36. Listening to talk shows and blogs. Just look at some the responses to this OP and you can see that

other people have this same impression. It is most likely because we don't really know what they are doing and what they are looking for and how they are going about searching for what we don't know.

[dysfunctional press]

24. recording them isn't the problem...

getting enough people to listen to/monitor them all IS.

[Lilith Velkor]

52. Exac-alack-ly.

[ThomWV]

25. That capability has been available for a long time, in fact far greater capability than that

They have had the ability to capture more information than that - and I really don't know how to say this. Just don't get it in your mind that capture of all things at all times is a longstanding capability; decades.

[serrano2008]

27. I agree. Not only is it not possible technologically but...

Less than .00001% of the information would be useful to them at all so listening to every American 24/7 is so inefficient that even the Govt. isn't that stupid.

Now...when you filter out what they're really looking for (terrorist info, anti-bush, bribe info), and all of the important people they may want to listen to (politicans, billionaires, journalists)...you're getting down to tens of thousands or even a few hundreds of thousands of people...and it becomes a little more possible.

Is the NSA listening to all of my grandma/grandpa's communications? No, that would be beyond stupid. Are they listening to the mayor of my city...well, probably.

[ThomWV]

39. You don't understand at all - technologically they have had this capability for decades

I know with absolute certainty that they had at least this much capability prior to 1970.

[Statistical]

40. They have had the ability to INTERCEPT not STORE.

One days worth of information exchange (internet activity, email, phone calls, fax, file transfers, etc) is MORE than the combined world storage capabilities.

Unless the US govt buys MORE storage than the entire WORLD combined (personal, business, govt) it is simply impossible to store that much data.

Even if they did the govt would only have storage space for a DAYS WORTH of communication.

To store just a few years worth of "everything" would require 1000x as much storage.

So world storage sales:
US Intelligence Agencies - 99.999%
Entire rest of world (all individuals, businesses, govts including US non intelligence govt services) 0.001%

Hardly.

The idea the US Intelligence Agencies STORE everything is simply impossible.

[Occulus]

45. Do you know and understand what 'metadata' is?

THAT'S what they're capturing, and I do believe they're capturing ALL of that. Yes, all. Every last bit- in the literal sense of the term 'bit'.

From this Kos diary:

Let's try the technical stuff first.

Mr. Tice talked at some length (on Olberman's show) about the difference between large-scale technical surveillance and more focused directed surveillance. If I've understood him correctly, then I think I can explain what he was talking about by using email as an example.

If you were interested in screening huge amounts of email, but didn't have the capacity to capture or store it all, you might decide to just content yourself with the metadata. Metadata is just "data about data". For instance, in the case of email, some interesting metadata might be: (a) what language it's in (b) the sender's address (c) the recipient's address (d) the length in bytes (e) the length in lines (f) what kinds of attachments, if any (g) what mail program was used to compose it (h) what the Subject line was, and so on.

This sort of metadata is relatively easy to extract and takes up a lot less room than the actual data: the metadata for an email message with 2M of photos attached might fit in 1K. (And this is the point where it should dawn on you that similar metadata exists for faxes, phone calls, and every other electronic form of communication.)

Metadata can be useful. Suppose you know that The Bad Guy always uses Eudora 1.1 to compose mail messages and always attaches photos that are 772x448 pixels in JPG format. If you've extracted the right metadata from billions of messages, you might be able to figure out that the 99.999% of them aren't what you're looking for by using that as a filter. If you're lucky, the only messages left will be the ones you want -- or the number will be small enough that brute force or maybe a simple search will get you what you want.

But metadata can be abused. It's possible to use the same collection to reconstruct the salient details of every message sent by A. Or from A to B. Or which has a "Subject:" line containing the string "protest". And so on. It enables ad hoc fishing expeditions that are limited only by the scope of the collection, the kind of metadata extracted -- and the restraint of those conducting them, which I think we can safely characterize as "nonexistent".

If I understood Mr. Tice correctly, metadata collection was untargeted and pervasive. They went for everything they could get. Which means if you sent a message to Aunt Mary with a photo of the dog on July 17, 2004, they acquired -- or at least tried to acquire -- the metadata for it.

Mr. Tice's further point was that high-level technical analysis like this was used to select specific targets for detailed analysis -- and in that detailed analysis, EVERYTHING was collected. Not just metadata: everything. Every phone call, every fax, every email, every instant message, everything. All captured and stored in a database...somewhere.

They don't capture entire recorded conversations each time they tap teh tubez because they don't need to. What they capture is who called whom, and when, and for how long the call lasted. They compare those notes among thousands upon thousands of calls and emails and faxes, and find out from that information who is socially networking with whom. In this way, the can indeed capture all of the relevant information without needing the exact conversation.

If necessary, I'll make this an OP, because it seems there are a lot of people under the mistaken impression that the technology to do this doesn't exist, when in fact it does, and has for probably decades.

[Statistical]

47. End result a massive expensive database mostly full of crap.

I work with databases every day.

Database = only a good as the information loaded into it.

Such a system where metadata but not actual content for EVERY conversionation, email, fax, post, transaction is saved would be 99.99999999999999999999999999999999999999999999999999999999% worthless.

As databases grow in size linearly the processing power to sort, filter, and generally make any use of it grows geometrically.

To not be selective would be to create the "worlds largest database of crap".

[hunter]

51. What about Google? How's that work again????

Seriously. If what you said was the least bit true then Google would be utterly useless.

The problem only grows "geometrically" if you demand 100% accountability.

If you allow a certain amount of information to bleed off as noise, then the problem remains quite manageable.

This is not the same as, say, a bank's database where you have to keep track of every last penny in every last account. Instead you can simply bleed off enough information to accommodate the size and power of your database engines -- as if a bank could just loosely round off everything to the nearest ten dollars and give you an answer, "yep, you've got well more than twenty dollars in your account so you can comfortably have a fast food lunch today."

You can even imagine an economic system where all the accounting is fuzzy. I think this is already the case in a certain sense. It might be impossible to get a clear accounting of Federal Spending even at a resolution of $100,000 or more. A certain amount money is bled off the books coherently, by corruption, and that's bad, but most of the lost information simply evaporates as noise to no particular person's or institution's benefit and we write it off (hopefully) to the general welfare of the nation.

[NeedleCast]

44. Storage and sifting is the issue, not intercept

[Bluenorthwest]

29. Actually I've not heard anyone say they recorded it all

I heard they 'monitored' 24/7/365, which is not the same as recording all of it. Maybe they did say that, but that is decidedly not what I heard.
NSA computers are not antiquated.

[bdamomma]

37. oh they have their ways.

[Occam Bandage]

38. I don't think that anyone with a modicum of intelligence has suggested

that the government records literally everything. Not only would that be enormously, preposterously expensive, there wouldn't even be any benefit.

Oh, as for computer systems? The NSA has the best in the world.

[hunter]

41. The data is very easily compressed.

There are huge redundancies in internet communications.

If someone downloads a video, for example, you don't have to record the entire video, all you have to do is record which video it was, where it came from, and where it went.

Let's say your typical terrorist video gets downloaded by 80,000 people. There's no reason to save 80,000 copies of the video, you only have to save one.

There's not so much original material on the internet as people think.

[JCMach1]

49. Absolutely they can do it... I knew someone in the house of spooks who worked

on the imaging side. The things (and resolution) in real time they were doing (mobile) in Iraq would boggle your mind.

That's all the detail I will give or I might disappear... no srsly

:hide:

[Uncle Joe]

50. Do you believe this would prevent the fascist/neocon types from trying, when they're in power? n/t

[kenny blankenship]

59. Not necessarily store it but scan it as it crosses the telecom - satellite network

And what the NSA has in supercomputers does indeed fill vast warehouse sized - arena sized-network operations centers. Crays are ancient history BTW.

A great deal of other data about you is being permanently stored - by helpful private financial corporations like TRW and other credit rating agencies, by banks that issue your checks and credit cards, by consumer profiling data mining market research outfits. Also internet service providers and search engine companies provide your data to the govt on demand.

It's not tin foil.