The two results chapters are starting to look fairly complete. I need to sit down and read over them soon, I imagine that will produce a lot of corrections and such.
Re-running sim-41 with a couple more things to set buffer size, not sure if it will change the result. I think I should be getting more goodput than I am. Looking at putting this result into thesis, it seems interesting enough.
Randomly found some interesting research: Lunar, which uses Weaves. Lunar gets the Linux networking stack, ports it to user space, and makes it available to the greater system for simulation purposes. Still haven’t got around to reading the thesis fully. Was finished off in early 2004. Weaves is pretty cool, it does some magic to automatically make programs reentrant at the linking stage or so – share code but not data sort of thing. Lunar makes use of this.
So way back in 98 or 99 Glomosim had FreeBSD 2.2.4 as a TCP model in the simulator. Neat. Not sure how I missed that until now.
They don’t make any fuss of it, it’s really just one or two sentences at most that note that they’ve got this TCP model. Sounds fairly important to me!
It’s becoming more apparent that one of the important facets of my work is that I have many TCP implementations.
The next chapter of my thesis will be about the performance variation between them. Yep, I’ve just put together the validation/accuracy chapter. It’s 21 pages long, 4300 words, 11 diagrams, 1 table, 1 source listing and cites about 36 different things I haven’t referred to earlier in my thesis (and a few I have).
Other random note: Thesis style regulations and guidelines
A bit more work today. More doing Intel stuff, but I did get some time on the thesis to work onwards, now on to the random loss scenario. I’ve described it a bit and added yet more citations. Need to update the graphs and analyse it. Before this I finished describing the SIP simulation, I think that is looking OK now. Wordcount is only about 3500 for this chapter now. Hopefully get the chapter to the “first draft” stage tomorrow, all going well. Would be nice to say I’ve done so in my weekly report…
Probably got another 800-900 words done today. Not quite like yesterday! But still, this chapter is coming together and looks a lot better with a bit of content. I’ve got more graphs in today which were integral and a couple of them look very good. Mostly hindered today by the bugs I hit earlier.
I’ve probably slowed down a little in my writing because I’m a little unsure of what I’m exactly saying with a couple of my simulations. Don’t quite have it clear in my head the argument I’m presenting. Should sort that out. Two more simulations to describe and update graphs for in the current chapter and then we’ll see what the state of it is. Is already long enough at 17 or so pages, so I expect it’ll be about 20 by the time I finish. Up to 3100 words in that chapter now.
Written up some more prose in my current chapter. Described the initial tcptrace graphs now.
I had to put in a couple of bugfixes into Linux 2.6 to do so. jiffies was going up to fast (!!!!). Also the simulated NIC driver had a small change to allocate skbuffs with a slightly different sizing scheme so the advertised window matched up with what we measured in emulation.
All my simulations have now finished.
Thing is, since I have more bugfixes, I really need to run them all again 
(now doing sim-9)
This seems to be the last lot of fixes for now (touch wood), as I have some Linux tcptrace graphs that are almost identical between simulation and emulation. Looking really good. I just have to find a nice way of presenting them in my thesis (should I use colour this time? If so, what do I do about the yellow? Really need to change the colour of the window line, but the only eps editing tool I have in Linux is inkscape, and it isn’t quite there yet to do this nicely. I used freehand in windows when doing so for my PAM poster).
Been a productive day. After my meeting with Tony on Skype last night, I felt inspired to do some work on the thesis.
I’ve split up the results chapter into an “accuracy of TCP simulation” chapter and a “variability of TCP implementations” (better title pending) chapter.
I’ve written about 2000 words of prose and added citations that I felt neccessary along the way (including Brendon’s 420). Also made a couple of diagrams and one new graph.
I now have 100 pages of main thesis, 16888 words, 116 citations. Not that many words yet, but the 2000 today helped! I know this writing will need a bit of work later, but I’m getting the ideas and words down there which is all good. I’m currently in the process of analysing traces for validation purposes. Its all very positive stuff at the moment. Hope to get another stack-full of words done tomorrow. Today that was about 7 or 8 pages worth of stuff that looks a lot better than it did before.
And set it running.
Looked over thesis to give myself an overall view of the document.
Added some more stuff. Looked at SCPS. It has a reference implementation.
Had a preliminary play around with kcachegrind and oprofile. Can’t use valgrind-callgrind on this machine because of misconfigured ubuntu/debian packes or so. I dunno.
Anyway, here are the top functions on a test of a few FreeBSD TCP streams in ns-2:
5.91 CalendarScheduler::insert
4.32 Packet::init
3.26 TclExecuteByteCode
3.08 get_stack_id
2.90 Scheduler::dispatch
2.48 Tcl_FindHashEntry
2.14 softclock
2.00 tcp_input
1.72 tcp_output
1.68 Scheduler::schedule
1.37 CalendarScheduler::deque
1.32 CBR_Traffic::next_interval
1.29 callout_reset
1.29 CalendarScheduler::head
1.25 .plt
1.21 TopoSort
1.05 soreceive
There isn’t anything that is hugely bad, the costs are fairly well spread out. As expected, inserting into the global event list is the most expensive operation. Packet::init can probably be made less expensive by remove all the packet headers we do not need. get_stack_id could just be an array index, not a function. Otherwise it is stuck there. The simulation was rather short, really need to run it a lot longer to remove the costly startup functions (I assume the Tcl stuff is mostly upon startup).
Anyway, looks like oprofile will be good for this in the future. Hope to use this desktop machine I have here to run a host of performance tests in the near future. Really need to sort out my results chapter first, but it makes sense to start getting all the performance results I need sooner rather than later.
Yep, I’ve got the simulations re-running again. Just got three simulation clients connected for now. Should check back on this in the afternoon or something.
Got (virtual) meeting tonight to discuss things. Need to organise what I’m going to talk about a bit more perhaps, it would help to have clear in my mind my current thoughts and problems.
I had set a whole lot of simulations to re-run with the goal of doing it over the weekend so nobody minded the machines being loaded.
Unfortunately there have been aircon/heating problems in the machine room, which means all of that has stopped. Guess I’ll have to do them over the week on the processing machine. Once the aircon is fixed.
Now thing about some graphs showing how the stacks respond to loss.
Also analysing the differences with MDAV, which I’ve updated a bit to work on my desktop at least.
OpenBSD in simulation has the exact same packet sequence for 39 packets. Then in emulation there is packet loss, in simulation there isn’t. In emulation there is a slight difference in timing, but I think things in sim are faster; so there should be packet loss in simulation too. And yes, the router queue size is the same in both. Seems a bit strange. Earlier on in simulation I was getting loss. Huh!
Still need to figure out how to present showing how similar the actual tcpdump-output-like packet traces are. I think I could improve on what I did in wintersim (just a bit of prose about how they looked).
I’ve just spent many hours debugging Linux 2.6 in simulation. I’ve referred back to emulation a couple of times. I’ve set up many tests, compiled my code more times than I could count, invoked gdb far too much.
I ended up putting many printf’s (or equivalent anyway) in the code and running again and again, wondering why:
- There are extra PSH flags on many packets in simulation
- The receivers advertised window didn’t grow like in simulation
I’ve previously put down 1. above to application models. This was in my wintersim paper. I eventually found that paper and saw that conclusion (it was with Linux 2.4) and thought “ok, that doesn’t matter, time to debug window size”.
I have done so for a few hours and now have a solution. Which is good.
The bad bit, is I have solved this all before and wasted many hours before, but for Linux 2.4. For some reason I didn’t port the changes across to my 2.6 tree. And the problem became very evident when I found it: the memory sizes for TCP buffers were configured with lower maximums. This was because simulation was effectively reporting less memory available than the emulation machine (which isn’t wrong per se, but not what we want for our test). Once fixed, the PSH flags are still there, but the advertised window is as expected.
The good news is, I can now have 3 sets of 2 graphs for sim/emu that match up really nicely.
NB: time to update, rebuild, and rerun simulations really! This might explain a couple of results where Linux26 wasn’t quite the same…
Not so happy with Linux26. From what I can tell, the difference is coming from the application. I can’t see any evidence to show that its anything to do with being in the simulation cradle at the moment. Not quite the result I was looking for.
Should maybe test Linux24. I dunno. I can’t quite figure it out enough to make myself happy. I theorise that the simulated and real app are pretty similar by now. Hmm, but not exact… I could update tcpperf to be more exact, and run a new emulation. Maybe worth a go. For now, I think I’ll head towards the gym. Been a while.
Update: Nope, using tcpperf with nonblocking mode made no difference that I can see with tcpdump… PUSH flags are still different. Hrmph.
Startup is looking really good now. I modified openbsd packet reception so the packets are received into the stack right away, rather than waiting for a netisr on a software interrupt (10ms timer). It is never quite obvious how this bit of the stack works: the driver -> stack interactions. I never quite know when the packet is actually supposed to be processed by the stack. And this functionality is very important to get very close behaviour in simulation. Anyway, can now make some nice startup graphs for openbsd that show it is super close.
FreeBSD and OpenBSD look really good.
Linux26 not so. Lots and lots of PUSH flags in simulation compared to emulation. Trying to track these down at least. de ja vu. I’ve been here before. Perhaps that was witth Linux24.
Compare packet traces and show these in the thesis nicely. Possibly use my graphing stuff. Investigate whether that is worth it.
Go through results chapter and decide how strong it is looking/going to look. See what conclusions I can draw from it. Start writing stuff in conclusions chapter. Refer back to introduction of problem statement. Sum this up, mail to Tony. Consider in light of making a paper.
Today my doctoral dissertation hit 100 pages, even though there is bugger all content in there. I thought I might put together some summary statistics:
- Total pages: 100
- Total number of citations on my citeulike: 148
- Number of citations current used in thesis: 95 (this includes web references, RFCs and ISO standards not included in my citeulike)
- Number of pages before main bit starts (title page, abstract, contents, etc): 11
- Figures: 9
- Tables: 2
- Code listings: 24
- Number of chapters, sections, subsections, etc.: 173
- Number of pages completed to “draft” standard: 31
- Number of bibliography pages: 7
- Approximate word count: 14815
Seems to limit queue size to 100! Bugger. Wanted some bigger queues than that. em-7 is useless because of this.
Oh well, so much for that experiment. On to the next. Not sure what it is just yet. I’m sure something will come up tonight.
Add sim-38 stuff to thesis…
Investigate em network, em-6 is finished.
Monday was spent working on simulation models and traces at Intel. Hopefully that work will turn into a decent paper in the future. Things look fairly positive, though there is quite a lot of work for others to do to make it happen. Deadline May 16 or so.
Just read over Brendon’s thesis. It is a little boring, but I think it is fairly strong and should well get him his Master’s degree. I’m somewhat envious really, I want to finish my degree!
Before I do that, I’ve got a bit more work to do. Why don’t I get back to it now and assess how much work I’ve got to do.