Interpreting the Data: Parallel Analysis with Sawzall
Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan
Abstract
Very large data sets often have a flat but regular structure and span multiple disks and machines. Examples include telephone call records, network logs, and web document repositories. These large data sets are not amenable to study using traditional database techniques, if only because they can be too large to fit in a single relational database. On the other hand, many of the analyses done on them can be expressed using simple, easily distributed computations: filtering, aggregation, extraction of statistics, and so on.
We present a system for automating such analyses. A filtering phase, in which a query is expressed using a new programming language, emits data to an aggregation phase. Both phases are distributed over hundreds or even thousands of computers. The results are then collated and saved to a file. The design -- including the separation into two phases, the form of the programming language, and the properties of the aggregators -- exploits the parallelism inherent in having data and computation distributed across many machines.
Published in:
Scientific Programming Journal
Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure 13:4, pp. 227-298.
Download: PDF Version
URL (Final): Journal link
Animation: The paper references this movie showing how the distribution of requests to google.com around the world changed through the day on August 14, 2003.
'Brain Trainning > 관심기술' 카테고리의 다른 글
간만에 보는 R 이제는 이쁜 UI까지 생겼네요 ㅎㅎ (0) | 2011.04.05 |
---|---|
MS가 꼽은 2011년 10대 IT 트렌드 [펌] (0) | 2011.01.13 |
First look: AirPrint wireless printing with Apple's iOS 4. (0) | 2010.11.10 |
Bigtable : Distributed Storage System (0) | 2010.08.23 |
The Google File System (GFS) (0) | 2010.08.23 |