1. Home
  2. Tutorials
  3. Clusters
Yolinux.com logo

List of Linux Filesystems, Clustered Filesystems, Performance Compute Clusters and Related Links

Links to sites covering Linux clustered file systems and Linux computing clusters. There are plenty of open source and commercial clustering solutions supporting Linux so that it will scale to supercomputer levels of computing and storage throughput.

Related YoLinux Tutorials:

°Linux Hardware components

°YoLinux Tutorials Index

Filesystems:

Clustered Filesystems:

Distributed Filesystems:

  • HDFS: Hadoop Distributed File System - distributed, fault tolerant storage for large datasets. Write and append distributed storage. Written in Java for MapReduce distributed computing clusters.
  • Ceph - POSIX compliant networked file system. Striped and replicated.
  • Coda File System - Advanced networked filesystem.

Logical Volume Manager:

High Performance Computational Clustering:

Linux HPC (High Performance Clusters)

ClusteringTools:

  • OSCAR - [github] - Installation, configuration, management, programming, and using clusters.

Scheduling:

Message Passing Interface (MPI): Programmers API for software to coordinate tasks access multiple nodes.

IPC Communication Libraries:

  • PVM (Parallel Virtual Machine):
  • TIPC: Transparent Inter Process Communication: intra cluster IPC. High speed fault tolerant and redundant synchronous sockets and asynchronous network communication.
  • UPC: Berkeley Unified Parallel C - C language extension. Uniform programming model for both shared and distributed memory hardware. Programmer is presented with a single shared Partitioned Global Address Space (PGAS) where variables may be read and written by any processor (SPMD: Single Program, Multiple Data).
  • Titanium: UC Berkeley developed Java "dialect" to support massively parallel supercomputers and distributed memory clusters.
  • OpenMP.org: Older cross platform API which supports shared memory and parallel programming in C/C++ and FORTRAN for NUMA and SMP Multi-Processing (MP) systems.

HA: High Availability Clusters

Linux Journal: A High-Availability Cluster for Linux - by Phil Lewis June 29, 1999

Grid Computing:

Distributed computing. Generally more loosely coupled than a cluster. A grid may be heterogeneous and geographically dispersed.

Cloud Computing:

Cloud computing infrastructure provides the ability to provision computing infrastructure, software, storage, security and data management as a service. Typically this is provided using distributed virtual systems preconfigured to perform these services. When greater throughput is required, more virtual machines are provisioned to support the load.

Commercial Cloud computing efforts are available as services purchased through vendors such as Amazon Web Services (EC2: Elastic Computing Cloud) or Google AppEngine.

Open source cloud frameworks are also available:
  • OpenStack.org - control large pools of compute, storage, and networking resources using a web dashboard
  • Eucalyptus.cloud - mimics the Amazon Web Services cloud and API
  • AppScale - provides compatibility with Google App Engine Applications
  • CloudFoundary.com - EMC VmWare supported project
  • CloudStack.org - Apache foundation sponsored. Java based. VM Hypervisors supported: VMware, Oracle VM, KVM, Xen.
  • Nimbus - Amazon EC2/S3-compatible. Targeted to the scientific community.
  • OpenNebula.org
  • OpenQRM - works with Debian, Ubuntu and CentOS Linux and VMware, Xen, KVM and Citrix XenServer virtual machines. Supported "Enterprise" edition available. N to 1 HA failover.
  • FOSS-cloud.org - Linux or MS/Windows - desktop support focus

Interconnect Hardware:

  • Myrinet - High speed low latency interconnection. Switch interconnect.
  • Dolphin Interconnect - Ring topology. Also 2D and 3D topologies. Traffic shared on ring. If one node goes out, the ring is broken and communications stop.

Infiniband:

High Performance / Low latency Gigabit Ethernet (GE):

Latency Comparison:

Hardware Latency (microseconds) Bandwidth
Quadrics 3 800 Mb/s (Elan 4)
Myrinet 6 800 Mb/s (Rev E)
Infiniband 4X (MPI driver) 6 1.8 Gb/s
(10 Gb/s Voltaire)
Level 5 Networks (Low latency GE NIC/driver) 7 end to end MPI
13 with switch
9 TCP end to end
Ammasso (RDMA) 16 end to end
Standard Ethernet (GE) TCP/IP 65 MPICH over TCP/IP
80-100 TCP/IP		 60 - 80 Mb/s
Source: Foundry Networks 2004

Network monitoring:

Links:

Book imageBooks:

"How to Build a Beowulf: A Guide to the Implementation and Application of PC Clusters"
by Thomas L. Sterling, John Salmon, Donald J. Becker, Savarese, Daniel F. Savarese
ISBN #026269218X, MIT Press 		Amazon.com
"Linux Cluster Architecture"
by Alex Vremios
ISBN #0672323680, Sams 1st edition 		Amazon.com
"Building Linux Clusters" (and CD)
by David H. M. Spector, Mark Stone
O'Reilly and Associates
ISBN #1565926250 		Amazon.com
"Server Load Balancing"
by Tony Bourke
O'Reilly and Associates
ISBN #0596000502 		Amazon.com
"In Search of Clusters"
by Gregory F. Pfister
ISBN #0138997098, Prentice Hall PTR

Theory and concepts.

Amazon.com
"Blueprints for High Availability"
by Evan Marcus, Hal Stem
ISBN #0471430269, Wiley

Amazon.com