IBM scans ten billion files during your lunch hour

Simon · Jul 22, 2011, 19:45:35

IBM has unveiled a single system that can scan 10 billion files in only 43 minutes.

If that doesn't sound impressive, keep in mind IBM's last record was one billion files scanned in three hours in 2007.

IBM made the achievement with its General Parallel File System (GPFS) - commonly used by top supercomputers - running on a cluster of ten eight-core systems paired with 6.8 terabytes of solid-state storage.

"GPFS's advanced algorithm makes possible the full use of all processor cores on all of these machines in all phases of the task (data read, sorting and rules evaluation)," IBM explained.

"The appliances sustainably perform hundreds of millions of data input-output operations, while GPFS continuously identifies, selects and sorts the right set of files among the 10 billion on the system," IBM said.

The breakthrough makes it possible to hold huge amounts of data on a single platform, while keeping it manageable.

Read more: http://www.pcpro.co.uk/news/368830/ibm-scans-ten-billion-files-during-your-lunch-hour

zappaDPJ · Jul 22, 2011, 22:23:23

But did they find the virus?

Simon · Jul 22, 2011, 22:40:34

"The"?

zappaDPJ · Jul 22, 2011, 23:05:35

Rik · Jul 23, 2011, 05:47:09

Where can I buy one?

Simon · Jul 23, 2011, 11:26:15

A virus?

Rik · Jul 23, 2011, 11:49:52

No, I can get them for free.

pctech · Jul 23, 2011, 11:54:38

Might be a tad big for your living room.

One of These might fit better.

Rik · Jul 23, 2011, 11:59:13

Being delivered on Tuesday.

Simon · Jul 23, 2011, 12:26:39

I note it doesn't say how much they cost. I'm guessing, if you have to ask, you can't afford it.

Rik · Jul 23, 2011, 12:29:06

Naturally.

Glenn · Jul 23, 2011, 12:33:56

From $25,000 is the lowest figures I can find with a quick search.

Rik · Jul 23, 2011, 12:36:14

In that case, I'll have two. Will it run XP?

Glenn · Jul 23, 2011, 12:37:07

The humble PS3 in clusters could be a cheaper option http://www.physorg.com/news149192580.html

Rik · Jul 23, 2011, 12:38:27

They'd get hacked too easily...

pctech · Jul 23, 2011, 13:24:17

The CX1 runs Windows HPC Server 2008