Diamond - Searching Terabyte Databases With Active Storage Technology

As part of a research project called Diamond, groups at Intel and Carnegie Mellon University are developing software for efficiently searching databases that can be as large as several terabytes (1 terabyte = 1,000,000,000,000 bytes). The system does not require search indices, but is based on active storage technology.

That is, it uses the storage devices that have considerable processing power themselves, so that much of the filtering can be accomplished without transferring the data to the machine that does the higher level processing. Furthermore, much of the processing can be done in parallel, as each active storage device performs part of the filtering independently.

The research team has developed two test applications to demonstrate the system: Snapfind for interactively searching for matching images in a collection of unlabeled photos, and VideoFerret for forensic reconstruction using video data.

The system runs on Linux and the source code is freely available for download. Read more...