Parallel session: supporting data-intensive applications

Editor's Notes

• #38: Each dataset might only be downloaded once Users might only want a small section of their data – over the course of an hour of data collection, perhaps only the last five minutes were useful Does not make sense to push every byte to a CDN I don’t know where my users are Some organisations have federated agreements with partner institutions –dark fibre, or VPNs over the Internet. I however have users performing ad-hoc access from anywhere.s I can’t block segments of the world – we have users from Russia, China etc Some scientists want to push data back to Diamond Reprocessing of experiments with new techniques New facilities located outside Diamond using our software analysis stack
• #39: When I started at Diamond, this is how users moved large amounts of data offsite. It’s a dedicated machine called a “data dispenser” it’s actually a member of the Lustre and GPFS parallel filesystems it speaks USB3 It’s an embarassment
• #41: We started testing Diamond’s own network by using cheap servers with perfSONAR, to run AD-HOC tests. First we checked them back-to-back that they could do 10Gb, then we tested deep inside the Science network next to the storage servers at the edge of the Diamond network, where we link to STFC Then we used perfSONAR’s built-in community list to find a third server just up the road at the University of Oxford to test against
• #42: Packet loss? Is that a problem? In my experience, on the local area, no, it's never concerned me before. This was the attitude of my predecessors also.
• #43: There are three major factors that affect TCP performance. All three are interrelated. The “Mathis” equation predicts the maximum transfer speed of a TCP link. TCP has the concept of a “Window Size”, which describes the amount of data that can be in flight in a TCP connection between ACK packets. The sending system will send a TCP window worth of data, and then wait for an acknowledgement from the receiver. At the start of a transfer, TCP begins with a small window size, which it ramps up in size as more and more data is successfully sent. You’ve heard of this before; this is just Bandwidth Delay product calculations for TCP Performance over Long Fat Networks. However, BDP calculations for a Long Fat network assume a lossless path, and in the real world, we don’t have lossless paths. When a TCP stream encounters packet loss, it has to recover. To do this, TCP rapidly decreases the window size for the amount of data in flight. This sounds obvious, but the effect of spurious packet loss on TCP is pretty profound.
• #44: With everything being equal, the time for a TCP connection to recover from loss goes up as the latency goes up.
• #45: The packet loss between Diamond and Oxford was too small to previously cause concern to network engineers, but was found to be large enough to disrupt high-speed transfers over distances greater than a few miles.
• #46: So... once we had an idea what the problem was, instead of just complaining at them, I could take actual fault data to my upstream supplier. Phil at STFC was gracious in working with us to allow us to connect up another PerfSonar server. Testing with this server let us run traffic just through the firewall, which showed us that the packet loss problem was with the firewall, and not with the JANET connection between us and Oxford
• #47: “Science DMZ” (an idea from ESnet) is otherwise known as “moving your data transfer server to the edge of your network” Your JANET connection has high latency, but hopefully no loss. If your local LAN has an amount of packet loss, you might still see reasonable speed over it because the latency on the LAN is so small. The Science DMZ model puts your application server between the long and short latency parts, bridging them. Essentially, you’re splitting the latency domains.
• #48: This section is cribbed from an utterly excellent presentation by Kate Petersen Mace from Esnet Science DMZ is about reducing degrees of freedom and removing the number of network devices in the critical path. Removing redundant devices, methods of entry etc, is a positive enhancement to security.
• #49: The typical corporate LAN is a wild-west filled with devices that you didn’t do the engineering on yourself: Network printers Web browsers Financial databases In my case, modern oscilloscopes running Windows 95… That cool touch-screen display kiosk in reception Windows devices In comparison to the corporate LAN, the Science DMZ is a calm, relaxed place. You know precisely what services you’re moving to the data transfer hosts. You’re typically only exposing one service to the Internet – FTP for example, or Aspera. You leave the enterprise firewall to do its job – that of preventing access to myriad unknown ports, and you use a more appropriate tool to secure each side.
• #50: Linux iptables will happily do port ACLs on the host at 10Gb with no real hit on the CPU.
• #51: This means that if a packet gets dropped in one stream, the other streams carry on the transfer while the affected one recovers.
• #52: This is how we implemented the Science DMZ at diamond You can see the Data Transfer node, connected directly to the campus site front door routers There is a key question on how do you get your data to your Data Transfer node? Move data to it on a hard drive Route your data through your lossy firewall front door – it’s low latency, so Mathis says this should still be quick We went a step further and put in a private, non-routable fibre link to a server in the datacentre. The internal server has an important second function – Authentication of access to data. I didn’t want to put an AD server in the Science DMZ if I didn’t need to, so this internal server is running the Authentication server for Globus Online from inside our corporate network
• #54: I tend to use perfSONAR just for ad-hoc testing. It’s fabulous for this, and I can direct my users to it for self-service testing of their network connections. I know it is very useful for periodic network monitoring, but I don’t take advantage of that yet
• #55: The common implementation of SCP has a fixed TCP window size. It will never grow big enough to fill a fast link. Aspera – UDP based system, commercial. never used it, I’ve heard some people from various corners of science talk about it, but I’ve never been asked for it. I’m sure it would do well on the Science DMZ if we ever need Aspera.
• #56: - globus online default stream splitting -> per file or chunks?