Validation

Validating NSC involves testing simulations against laboratory setups. The Wand Network Research Group have a network of computers configured for this purpose called the Wand Emulation Network.

FreeBSD and Linux have had thorough validation via direct comparisons with measurements recorded from the emulation network. Some testing has also been done for OpenBSD, though to a lesser extent. During this testing some bugs and strange TCP behaviour have been discovered.

The validation work has found some interesting results of its own which is featured as a poster at PAM2005.

FreeBSD

The laboratory network was set up with software routers configured with Dummynet to limit bandwidth and add latency. The topology was tested with ping and iperf, then a simulation script was made of the same setup. Tests were performed with iperf and tcpperf on the laboratory network and measurements were taken in simulation. Packet traces could be produced from both and compared using tcpdump and tcptrace.

Because FreeBSD's was the first network stack integrated into the simulation cradle, the methods used to move it were not as robust as those which followed for the subsequent stacks. Hence some time was spent debugging why the timing of packets was not as consistent as it should have been between simulation and laboratory tests. Around a 10% difference in goodputs was found: simulation was quite far behind. The sequence of packets produced was the same in both, however. Because the sequence of packets was exactly the same it was possible to produce a graph showing packet number versus time difference.

The following graph was produced that shows the time for every packet sent and received in a laboratory test minus the time for the same packets produced in simulation. The final green line shows the desired result, a nearly straight line. This means that not only the exact same sequence of packets was produced in simulation and the laboratory test, but the timing of the packets is very close also (millisecond accuracy was achieved).

Time difference vs. Packet number

The output of tcptrace comparing a real trace with one generated from simulation follows:

Real vs. simulated FreeBSD

Linux

Linux has relatively few problems, the mistakes made with FreeBSD were not repeated, so the validation process was a lot quicker. The following image shows output from tcptrace comparing a trace measured on the emulation network to a trace generated from simulation. The only noticeable difference is the PUSH flag is set in different packets. This is believed to be because the model of the application in simulation is not quite the same as the real application.

Real vs. simulated Linux

This result can be analysed further by hand by comparing the textual output of tcpdump. Doing so showed that the packets generated in both cases are almost identical, with only slight timing differences and the aforementioned difference of the PUSH flag. The above image shows some interesting facets of Linux's TCP implementation such as the dynamic receivers advertised window that grows throughout slow start.