I noted that the HZ value used in simulation and emulation did not match. FreeBSD defaults to HZ=100 (at least for 5.3), and I wanted to use this default for endhosts. So I made an image with a kernel compiled for that HZ value.
When I have the bandwidth window enabled (the default), there are differences between simulation and emulation (the previous test had this disabled, and things were very close). At first I thought this was because of the different value of HZ, resulting in different timer granularities.
That does not seem to be the problem. There is a useful sysctl net.inet.tcp.inflight.debug, which I set to 5. It outputs information like the following (from emulation):
0xc15c6a98 bw 1 rttbest 200 srtt 160 bwnd 2896
0xc15c6a98 bw 25527 rttbest 185 srtt 183 bwnd 4363
0xc15c6a98 bw 46600 rttbest 185 srtt 225 bwnd 5881
0xc15c6a98 bw 82067 rttbest 185 srtt 222 bwnd 8102
0xc15c6a98 bw 120928 rttbest 185 srtt 209 bwnd 10340
0xc15c6a98 bw 108998 rttbest 185 srtt 230 bwnd 9946
0xc15c6a98 bw 129268 rttbest 185 srtt 239 bwnd 11460
0xc15c6a98 bw 113034 rttbest 185 srtt 275 bwnd 11020
0xc15c6a98 bw 111531 rttbest 185 srtt 262 bwnd 10668
0xc15c6a98 bw 104050 rttbest 185 srtt 258 bwnd 10081
0xc15c6a98 bw 112422 rttbest 185 srtt 242 bwnd 10379
0xc15c6a98 bw 102807 rttbest 185 srtt 257 bwnd 9996
0xc15c6a98 bw 111089 rttbest 185 srtt 251 bwnd 10463
0xc15c6a98 bw 111494 rttbest 185 srtt 220 bwnd 9934
And from simulation:
0x868285c bw 12783 rttbest 179 srtt 184 bwnd 3619
0x868285c bw 51060 rttbest 179 srtt 218 bwnd 6055
0x868285c bw 80704 rttbest 179 srtt 219 bwnd 7914
0x868285c bw 122507 rttbest 179 srtt 187 bwnd 9901
0x868285c bw 155768 rttbest 179 srtt 186 bwnd 11755
0x868285c bw 204162 rttbest 179 srtt 177 bwnd 14252
0x868285c bw 231483 rttbest 179 srtt 197 bwnd 16495
0x868285c bw 199080 rttbest 179 srtt 183 bwnd 14156
0x868285c bw 209239 rttbest 179 srtt 197 bwnd 15188
0x868285c bw 275490 rttbest 179 srtt 185 bwnd 18564
0x868285c bw 214277 rttbest 179 srtt 206 bwnd 15752
0x868285c bw 191283 rttbest 179 srtt 181 bwnd 13655
0x868285c bw 209415 rttbest 179 srtt 184 bwnd 14741
0x868285c bw 231281 rttbest 179 srtt 195 bwnd 16411
Let us concentrate on the first variable, bw, first. This is the simplest to analyse. The code to figure this out is below:
save_ticks = ticks;
if ((u_int)(save_ticks - tp->t_bw_rtttime) < 1)
return;
bw = (int64_t)(ack_seq - tp->t_bw_rtseq) * hz /
(save_ticks - tp->t_bw_rtttime);
tp->t_bw_rtttime = save_ticks;
tp->t_bw_rtseq = ack_seq;
if (tp->t_bw_rtttime == 0 || (int)bw < 0)
return;
bw = ((int64_t)tp->snd_bandwidth * 15 + bw) >> 4;
tp->snd_bandwidth = bw;
Basically, we first set bw to (amount of data received / time) then merge it with tp->snd_bandwidth, making it into a long term average. So if you are receiving more quickly, the bw will go up to reflect this and tp->snd_bandwidth will move slowly upwards. It’s just a bandwidth estimator. Pretty simple stuff.
Unfortunately printing the debug info out (since I was connected via serial console!) slows emulation down and biases the result heavily. We can see that emulation has lower numbers that simulation, which matches up when the debug output is enabled. Normally emulation goes faster than simulation.
RTTs are a bit lower in simulation, due to ideal conditions I guess. I’m not sure, there is still something different in the simulation and emulation test. More analysis required.
