So I’m having to re-run a few experiments to sort out some results for my thesis. Not too many need to be re-run, just a few that I didn’t save all the results for or something silly like that. Also one or two that I didn’t get conclusive enough results from.
Emulations: em-6
Re-running emulations and have been doing so for a week or two by now. Unfortunately I had stuffed up my emulations twice so I didn’t get any useful results. First of all copying files off machines would randomly pause indefinitely, then second I realised I had stuffed up the router config so I didn’t do any traffic shaping, completely destroying the point of the experiment.
The second problem was solved by putting the correct IP address ranges into the IPFW config. My emulation scripts are a bit simplistic and they didn’t know how to figure these out for themselves.
The first problem required writing some Python code to kill off the copy process if it took too long. I suspect there is a bug elsewhere in the software stack I am using, but I really don’t have time to investigate this, so killing the copy if it stops seems like a reasonable approach. The copy always seems to copy all the data, just sometimes (maybe 1 time in 5 or so) it doesn’t close the connection. I think this is specific to OpenBSD. While it would be easy to blame OpenBSD, I think it much more likely some of mine or Brendon’s software which is being used here is very dodgy. Anyway, there isn’t a nice way to wait on process completion with a timeout like in Windows (WaitForSingleObject anybody?). I hacked up a nasty polling solution; I should perhaps really have investigated adding a SIGCHLD handler or using SIGALRM. My way is good enough for the situation and the tests now seem to be running fine. Got about 22 completed of 900 so far; might take a week or so for them to finish.
Performance simulations: sim-37
I had to build ns-2, NSC on my home desktop. Didn’t seem to have these set up there. I was getting a weird error from SCons until I realised it doesn’t install builders for e.g., CXXFile until you actually install flex! So I installed all the software I needed (gcc-3.3, gcc-3.4, g++-3.3, g++-3.4, flex, bison, libpcap) and built all required software.
After setting my LD_LIBRARY_PATH it seemed to run OK. Then I found that I needed malloc_debug.so, which I checked out. It was renamed to heapprof.so a while back. I updated my scripts to use the new name and new path to ns-2 and ran them. First, it died at the end of a long simulation due to me not patching the ns-2 FullTCP agent with my minor change that counts the amount of bytes received in a 64-bit int. Second, it died during shutdown due to a SIGFPE when deallocating memory at process shutdown.
I fixed a couple of bugs in heapprof.so which led to this and it got a bit better. Unfortunately this still seems to happen for some processes (not all). I’ve checked in some bugfixes but I really can’t figure out what is going on now; it looks like we are simply freeing valid memory that was allocated earlier; yet we get a SIGFPE. Unfortunately our test app, “sieve”, works fine. The setting of the env var MALLOC_CHECK_ affects things here. I just don’t have time to investigate this right now.
So I’ve turned off the memory sims and I’m just running the CPU sims right now. Still haven’t quite decided which ones I actually need to re-run; for now I’ve just started them all and I’ll figure out which is important later.
Weka and sim-34
Also installed Weka so I could regenerate the stats I was trying to report. This went smoothly, with the exact same figures as last time. I have now saved all the output in Subversion — can’t believe I forgot to do this last time! It did take a while to figure out how I generated the second set of difference data for Weka, but after staring at the screen for a while I realised it was very simple. I put some very simplistic code to generate this into a script and put that in Subversion as well.
Just need a bit more info on attribute selection stuff. Need to justify that the Information Gain attribute selection algorithm is good enough for us? Seems to be a fairly “de facto standard” attribute selection algorithm to me, but then, I don’t really know much about this area.