Systems We Make

[...] optimizations such as reducing the number of disk seeks to read a BLOB to 1 almost always, fault tolerance through replication (replication factor of 3) across geographies and many more such [...]

[...] it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel’s unique combination of [...]

[...] as in the number of compute units required to perform an analysis task should be low. 2. Fault tolerance in the context of analytical workloads means that the query when distributed for processing [...]

[...] distributed storage systems. Writers contact the metadata server to find out where to write a new block; the metadata server picks a data server, durably [...]

[...] the query and then forwards the execution plan to the master node in the worker pool. The metadata server provides an API to store and fetch metadata such as table names and schemas, and pointers to [...]

[...] engine and the query serving engine of this OLAP system. Another newness is the use of a key value store, Voldemort, to fetch the results of queries. Contrary to conventional designs where the cube [...]

[...] of the game represent the applications that access the information persisted in this key value store. By virtue of the role they play each of these participants can naturally tolerate [...]

[...] stamps through replication. A Partitioned Namespace for Objects The objects in this storage system are all part of a single global namespace. This is achieved by using DNS and [...]

[...] Reads : This form of guarantee is something that is offered to a single client of the storage system. It assures the client that the returned value is either the same across multiple read calls [...]

[...] Haystack was the primary storage storage system designed initially for Facebook’s Photos application. Its been around for almost 7 years now. [...]

Abstract : We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability. Ceph maximizes the s [...]

While we usually see enough papers that deal with the applications of the Map Reduce programming model this one for a change tries to address the limi [...]

Abstract HP’s StoreAll with Express Query is a scalable commercial file archiving product that offers sophisticated file metadata management and se [...]

[...] overview F1 is built on top of Spanner. Spanner offers support for for features such as – strong consistency through distributed transactions (2PC), global ordering based on timestamps, synchronous [...]

[...] the different storage abstractions, provides a namespace for them, transaction ordering and strong consistency for the objects and caching services. Front End Layer : This layer is a stateless server [...]

[...] . Intuitively, we can describe two extreme approaches to ensuring consistency. Namely, strong consistency and weak/eventual consistency. To make things more interesting researchers have formulated [...]

[...] ) Synchronous replication and automatic failover 3) Strong transactional semantics Link to the original paper [...]

[...] -row transactions in HBase. Seems to be motivated by Google’s Percolator. Link to the original paper [...]

[...] metadata management, supporting more than 250,000 metadata operations per second. Link to the original paper [...]

[...] . In some ways WAS takes on responsibilities very similar to what we see in a distributed file system’s name/metadata node such as data placement across disks, replication and load balancing. [...]

[...] ’s state is persisted in a set of B tree like files and a write ahead log on a distributed file system called Colossus (the next gen of GFS). Replication is provided by the spanservers by [...]

[...] becomes important to understand the consistency model that a distributed data store (be it a file system or a NoSQL or SQL database) offers. Most cloud storage systems make this choice for us and a [...]

[...] with custom metadata fields. The product brings together two significant systems: a scale out file system and a metadata database based on LazyBase. In designing and building the combined product, we [...]

[...] Stamp is a cluster of multiple racks. Typically there are about 10-20 racks with about 18 storage nodes per rack. Currently each storage stamp holds about 2PB of data but is soon expected to scale [...]

[...] . Just the kind of system I would love to cover for this blog! Data is stored on dedicated storage nodes, called tractservers (comparable to HFDS name nodes). The tractserver is a network front-end [...]

[...] very similar to what we see in a distributed file system’s name/metadata node such as data placement across disks, replication and load balancing. At a high level it is logically organized into [...]

[...] mappings. A directory is a collection of keys that have a common prefix. Its also a unit of data placement. All the data within a directory have the same replication configuration. Spanner moves data [...]