Home
Publications
Awards
Research
NB Collectives
MPI Topologies
MPI Datatypes
Netgauge
LogGPS
OS Noise
eBB
Network Topologies
Ethernet BTL eth
ORCS
DFSSSP
Older Projects
cDAG
LogGOPSim
CoMPIler
Teaching
Miscellaneous
Full CV [pdf]
BLOG
Events
Past Events
|
Netgauge - A Network Performance Measurement Toolkit
Netgauge Description:
Netgauge is a high-precision network parameter measurement tool. It
supports benchmarking of many different network protocols and
communication patterns. The main focus lies on accuracy, statistical
analysis and easy extensibility.
|
|
Download Netgauge:
See README file in package!
|
|
Building Netgauge
If something is wrong with the high-precision timer (e.g., frequency
scaling invalidates the results), MPI_Wtime() can be selected at
configure time by passing "HRT_ARCH=6" to configure!
On Linux Systems:
- ./configure MPICC=mpicc MPICXX=mpicxx # set appropriately
- make
- mpirun -n 2 ./netgauge (-- help)
On Cray XT-4/5 (tested on Jaguar@ORNL):
- ./configure MPICC=cc MPICXX=CC
- make
- aprun -n 2 -N 1 ./netgauge (-- help)
On Blue Gene/P (tested on Eugene@ORNL):
- ./configure HRT_ARCH=3 # needs override because cross-compilation can't execute
- make
- submit job to batch system
On AIX and Linux/POE (tested on BluePrint@NCSA and BlueDrop@NCSA):
- edit llscript.ll for your setting (if necessary)
- export MP_LLFILE=$(pwd)/llscript.ll
- ./configure MPICC=mpcc CFLAGS="-qlanglvl=extc99" MPICXX=mpCC ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes HRT_ARCH=6 # inline assembler not gcc compatible
- make
- submit job to batch system
|
|
|
Netgauge is developed in the Computer
Architecture Group at TU Chemnitz and the Open Systems Lab
at Indiana University. Many students contributed to
netgauge, see the "Authors"-file in the distribution.
|
|
Netgauge's Main Goals
- enable portability (a single tool measures many networks and communication patterns)
- avoid common benchmarking problems (pipeline-effects, influence of outliers to the average values)
- enable detailed measurements (more than rount trip time (RTT) and bandwidth
- parametrize network models
- offer a simple and extensible framework to implement different ideas
- combine efforts of network developers (nwho write network modules) and application scientists (who are interested in communication patterns)
- offer a portable high-precision timer
- support many networks and patterns "out of the box"
|
Netgauge's Patterns and Modules
The Netgauge framework consist mainly of commmunication patterns that
implement benchmarks and communication modules that implement low-level
network drivers. Other parts are the netgauge framework itself which
interact with the user to read and process command line arguments and
passes them on to the modules and patterns. A statistics and output
framework can be used by the pattern implementor to easily achieve
common tasks (e.g., get average/median or print data to a file). The general framework is shown in the following figure:
Patterns
The variable P is used to represent the number of nodes in MPI_COMM_WORLD!
- one_one - simple ping-pong patterns, if started with more than 2 processes, half of the processes (P/2)communicate with the other half in a ping-pong scheme
- 1toN - one to many communication scheme, a randomly chosen root communicates data to P-1 processes and they send it back
- Nto1 - one to many communication scheme, P-1 processes communicate data to a randomly chosen root it sends it back
- meas_o - measures the overhead in the pLogP model as described in "Fast Measurement of LogP Parameters for Message Passing Platforms"
-
- loggp - measures LogGP parameters with the method described in "Low-Overhead LogGP Parameter Assessment for Modern Interconnection Networks"
Modules
Different communication modules for many networks are available out of the box. Many of them are experimental which means that the implementation is not well tested. Netgauge is a research tool, so users should expect errors
(patches are generally welcome).
Two Sided Protocols
- MPI - Message Passing Interface (MPI) communication
- TCP - Transmission Control Protocol (TCP) comunication, implemented with standard POSIX sockets
- UDP - Unreliable Datagram Protocol (UDP) communication, implemented with standard POSIX sockets
- ETH - low-level Ethernet socket communication. Using RAW sockets (Linux) and forging customized Ethernet packets
- EDP/ESP - Ethernet Datagram/Stream Protocol implementation (experimental)
- IB - experimental InfiniBand implementation
One Sided Protocols
- MPI2OS - experimental MPI-2 One Sided implementation
- ARMCI - Aggregate Remote Memory Copy Interface implementation
The internal interface of Netgauge is two-sided. Thus all one-sided
operations are "mapped" to two-sided operations. The scheme is depicted
in the following figure. The module is supposed to issue the one-sided
data-transfer and after that increment a counter on the remote receiver
side. The network has to guarantee that those two operations do not pass each other! Both partners keep track of the appropriate peer's
counter values.
|
|
Examples
IP over IB (ofed 1.1) compared with Open MPI 1.1:
|
References
| HPCC'07 | [1] Torsten Hoefler, Torsten Mehlan, Andrew Lumsdaine and Wolfgang Rehm: | | | Netgauge: A Network Performance Measurement Framework Vol 4782, In Proceedings of High Performance Computing and Communications, HPCC'07, presented in Houston, USA, pages 659-671, Springer, ISBN: 978-3-540-75443-5, Sep. 2007,   |
|
