1. Introducing the Grid...

Before we can understand the motivation behind GridMon, we must understand a little about "the Grid".

The Grid can be a difficult concept to describe, but it is essentially a means for sharing resources (processing power, databases, radio telescopes...) via a distributed computer network.

Importantly for GridMon is that Grid enabled projects are expected to be large bandwidth users. A common example of the application of Grid technology is processing the data generated by new generation HEP experiments. These experiments are expected to generates large amounts of data, which will require fairly intense post-experiment processing. In this case, the Grid would facilitate the splitting of the experiment's dataset into small "chunks", with these smaller sets of data being transferred to multiple, distributed sites for processing. When each site has finished processing its share of the data, the results are combined. Some may say that given today's computing and networking technology, that this is not a overly difficult process. If we say however, that the amounts of data produced are expected to be in the order of Petabytes (1 Petabyte is approximately 1,000,000,000 Megabytes) then this will hopefully highlight the scale of the task to be undertaken.

2. Purpose of the work

“	...design and deploy an infrastructure for
	network performance monitoring within
	the UK e-Science community.	”

Since UK e-Science will be big users of Grid technology, the work is essentially to provide end-to-end network performance monitoring for the UK e-Science Grid(s), and has four key aspects:

• Publish results to Grid middleware - Results of the monitoring (network metrics) will be published to the Grid middleware, and via the middleware to end user applications.
• Visualisation for humans - Performance data will be also published to humans: typically network support staff and end users.
• End-to-end - By end-to-end performance, we mean performance as seen by Grid applications and end users, and not the view of the network itself. Monitoring may show that core network performance is good, but this does not necessarily correlate with local performance and hence end user experience.
• Ability of TCP wrt high b/w networks - some work will also be conducted to look at how well (or not) TCP works with high bandwidth networks.

3. Why?

We monitor the performance of networks to identify faults and inefficiencies. Experience shows that it is better to find faults before they find you! We also monitor to get an idea of the performance we can expect.

Some people believe that our understanding of network performance monitoring has reached maturity, and would therefore likely question the fuss being made over the Grid. In answer, regardless of current network understanding, monitoring for the Grid isn’t as clear cut, quite simply because Grid networking is different to what has gone before:

• data is transferred in large blocks (e.g. large TCP window sizes)
• transfers can be performed using multiple streams (multiple connections)
• end-to-end performance is now of great interest

A crucial aspect of this monitoring is the addition of non-human consumption, the idea that the middleware will take the results of the monitoring, and use them. Take the example of routing: if a file we require is located at Manchester and London, and based on recorded performance data the middleware knows that the connection to Manchester is faster and more reliable than that for London, then we would obviously wish to obtain the file from Manchester.

End user applications will also be able to obtain network metrics (from the middleware) upon which they can make decisions concerning their own communication (e.g. the number of streams to use and TCP parameters such as window size).

4. What?

Do we need a definition of metrics?

The metrics measured by GridMon are:

• Connectivity
• Jitter
• Packet loss
• RTT (Round Trip Time)
• TCP & UDP throughput

Their relevance to network performance is summarised below:

Packet loss
Data is transmitted in packets, and unsurprisingly, packet loss is an of measure how many packets are lost during transport. This includes packets which are discarded because they have arrived with corrupted "transmission" data (separate to the payload/user data part of the packet).

Packets that are discarded or fail to arrive must be re-transmitted, and this quickly causes a "traffic jam" if the packet loss is severe.

Connectivity
Is simply an indication of whether you can connect to a remote site/machine, and is closely linked to packet loss. It can be used to identify sudden faults, such as link failures, or more sporadic problems such as loss of connectivity at certain times of the day, perhaps as a result of network congestion occuring during "busy" periods, causing packets to be discarded.

Inter-packet Jitter
Is a measure of the variation in the delay of packets arriving. It is very important metric for real-time applications such as video conferencing, which require packets to arrive at a steady rate.

This metric is currently only measured for UDP traffic (including multicast) but it is planned to extend this for TCP traffic in the future.

RTT
Round trip time is a measure of the time it takes to send a packet from node x to y, and receive a response back at x.

TCP is a send-acknowledge protocol. A block of data cannot be sent until the receipt of the previously transmitted block has been acknowledged. If this acknowledgement takes a long time to arrive (due to a long RTT) then transmission delays are created.

TCP/UDP Throughput
Throughput is essentially a measure of the rate at which data is or can be transferred. However you must be careful about what kind of throughput you are talking about. For example, are you talking about maximum throughput, network throughput, end-to-end throughput, or throughput on the wire?

The GridMon toolkit monitors network throughput (what the network sees) for TCP and UDP traffic, and end-to-end throughput (what end user applications, and hence human end users see) for TCP traffic.

5. How (1)?

Figure 1 - GridMon Monitoring Architecture

The monitoring is performed by a kit of tools installed on a suitable machine at each e-Science Centre. Performance data is stored locally on that machine, and is published to interested people via a web interface, and to the Grid middleware via a publication service (LDAP, Grid or web). LDAP seems to be popular in the States, with OGSA growing in popularity in Europe. An (OGSA) Grid service is essentially a web service with some Grid specific add-ons/pre-requisites.

Every 30 minutes (90 minutes for bbcp/ftp) each machine performs monitoring between itself and all other e-Science Centres. In this way a mesh of monitoring is created, allowing each centre to build up a picture of the quality of its links to all other centres. The mesh approach is feasible given the relatively low number of sites involved (12-15 in this case). The times at which the individual machines run tests are staggered in an attempt to minimise the disruption one machine’s tests may have on another’s.

The monitoring host must obviously meet some requirements, with the most important being that:

• It is a dedicated monitoring machine. There is little point installing the toolkit on your web server, for example, because it will not allow like for like performance data to be connected.
• In the same vein, the machine must have similar connectivity to the other networked e-Science resources at your centre. Performance data will not be representative if the tools are, for example, installed on a machine hanging from the primary link to outside world, while users in the e-Science centre are three or four levels further down the network hierarchy.

5. Monitoring: How (2)?

Metric	Tool	Origin
Connectivity Packet Loss RTT	PingER	ICMP Ping SLAC et al
TCP thruput	IperfER	NCSA's Iperf SLAC and UCL
	bbcp	SLAC
	bbftp	IN2P3 = Tool SLAC = monitoring
UDP thruput	UDPmon	HEP@man.ac.uk, EDG
Mulicast thruput packet loss jitter	MiperfER	IperfER MCC

PingER is a collection of Perl scripts which use the ICMP ping utility to send 10 ping requests to remote hosts. The results (RTT etc.) are recorded for later analysis. The tool was developed at SLAC, with assistance from various other sources.

IperfER is based on NCSA’s Iperf utility, used for measuring the network’s view of TCP or UDP throughput between two hosts. Iperf consists of client and server executables which sit at either end of a TCP/UDP connection, streaming data between each other. IperfER allows the findings of Iperf to be stored for subsequent review.

UDPmon is essentially a UDP equivalent of IperfER, developed in Manchester University’s HEP department, within the bounds of the EDG project.

bbcp and bbftp are basic tools for copying files between sites, albeit using multiple TCP streams and large TCP window sizes. They are the important in relation to monitoring because they are end user software, not monitoring tools. This allows us to obtain a picture of end-to-end performance. This approach has been pioneered at SLAC. Building on their work, the monitoring toolkit will use SSH to login to a remote machine (via RSA public-private key authentication) and copy a series of files (ranging in size from 1 to 80 MB) across the network. Data about the transfer is recorded for future processing. GridFTP is another option.

MiperfER is a (new and experimental) multicast version of the IperfER tool. A common application for multicast is video conferencing, of which Access Grid is a well known e-Science example. It is also proving useful for e-Science meetings, hence GridMon's interest in multicast, and as the Grid promotes the use of geographically distributed teams (via VOs) it is expected that the demand for multicast applications (like video conferencing) will increase further. Note however that MiperfER is intended for use as a diagnostic tool, and that any benefits of publishing data to the middleware are unclear.

6. Benefits vrs Costs

Benefits	Costs
Grid software will make decisions based on network performance "right" metrics = better decisions Invaluable tool in detecting problems, e.g. bottlenecks Users can see performance for themselves	Bandwidth testing is intrusive Machine to host tools must be made available

The bandwidth testing (bbcp/ftp, IperfER and UDPmon) is undoubtedly intrusive. You cannot transfer data across a network without it having some affect. However, it is essential to transfer real data across the network to gain a more realistic picture of network performance. This cost is also far outweighed by the benefits:

• The Grid middleware and applications are going to make operational decisions based on previous network performance. As use of the Grid increases, bandwidth will become more scarce, and as a result decisions made by the middleware and applications will have pronounced consequences. If the wrong decisions are made, it will cause pain. Having the ‘right’ metrics available will allow ‘better’ decisions to be made.
• As the example plots in this document show, faults and inefficiencies can be identified.
• For the first time, end users will be able to see for themselves what performance they should be expecting from their Grid applications, regardless of what they know (or think they know) about the speed of the network connections used by their site or individual computer.

The cost of a monitoring host, nominally a PC, is negligible relative to the cost of a centre’s other e-Science equipment. And given the level of investment in network infrastructure undertaken by most sites, it would be a false economy to simply assume optimum performance was being achieved. Looking at it another way, would you attempt to service a Ferrari with a toolkit given away free with 20 litres of petrol?