A paper from Microsoft on data center traffic patterns.
It measures traffic from socket level. From the data, the following are found:
- In general, data center traffic is more regular than ISP networks, due to tighter coupling between different components
- Because data center applications prefer placing jobs with high traffic exchange close to each other for higher bandwidth, there is a work-seeks-bandwidth pattern, i.e. high intra-rack or intra-server traffic
- There are frequent one-to-all or all-to-one traffic, correspond to the map and reduce operation. This is a scatter-gather pattern.
- The traffic between servers are heavy-tailed. Servers within the same rack are more likely to exchange more traffic
- Median number of correspondance for a server is: 2 other intra-rack server, and 4 servers outside the rack
- If define congestion as 70% of link speed, among the 150 inter-switch links, 86% of links observe congestion lasting more than 10 sec in a day and 15% links observe congestion lasting more than 100 sec in a day. Most congestion is short-lived
- If a job sends traffic over a congested link, they are 110% more likely to fail to read the input
- Reduce jobs are responsible for a fair amount of traffic. Extract jobs in Dryad as well, which converts raw data into input stream
- Defining a flow as five-tuple flow, most flows are short-lived (80% flows are less than 10s), and very few are long running (less than 0.1% last longer than 200s)
- Traffic changes quickly
- Several times a day, all used links run close to capacity
- TCP incast is not observed. Number of contending flows are small as well. Reasons could be, (1) the number of open connections are limited, (2) local communications such as intra-server or intra-rack are preferred.
- A potential problem: Due to small BDP, window of TCP are small. A few packet drop will easily make TCP timeout.
Bibliographic data
@inproceedings{
title = "The Nature of Datacenter Traffic: Measurements & Analysis",
author = "Srikanth Kandula and Sudipta Sengupta, Albert Greenberg, Parveen Patel, Ronnie Chaiken",
booktitle = "Proc. IMC'09",
month = "November 4-6",
year = "2009",
address = "Chicago, Illinois, USA",
}