Sam’s Network Simulation Cradle Blog

Running more distributed simulation. As in, I have 99 machines running simulations at the moment.

Starting up machines to run simulations isn’t very nice, I have what I term a “hax0r” way of doing it.

I used autoexpect to record an expect script to start a simulation client earlier in the process.

I’m sure I could come up with a nicer automated way of doing this, but hey, this has worked a charm so I see no reason to spend any more time on it for the time being.

I’m beginning to think I want to bin/discretise my data and make a histogram of the goodputs achieved for each stack. Then compare these histograms side by side to see the differences. This should be rather interesting going on the jittered scatter plots I’m looking at at the moment.

Ok, I’ve done this, or am in the process of doing this.

I see that the distrobution of goodputs achieved for each stack is quite different. The “real” stacks especially are very different in their shape. The data needs to be made a bit more accessible, it is a bit hard to compare right now.

Also need to look at how the stack used to congest the data effects e throughput. From looking at the data in weka in the visualiser, it seems that the real world stacks are quite different here; perhaps they are more aggresive.

Actually, looking at this some more does not seem to confirm what I was seeing in the visualiser. Hmm.

There are times when gnuplot doesn’t really cut it. Mainly when dealing with things that should be more histogram-like in nature. It does have some support for this, but it isn’t the greatest.

There is ploticus, which I think I need to know a bit better. It has a bigger learning curve and looks a bit ugly when run by default, but seems produce good enough graphs if you play around with it enough. And it is a LOT more flexible than gnuplot.

Alternatively, you spend time coercing your data into a format that gnuplot diplays ok. It wont be good, but it will be easy enough. This is what I’ve done so far.

So I’ve looked at some graphs of the data for sim-33. I’m currently leaning towards feeling that the differences here are quite fundamental. It seems that many of the parameters don’t vary the results tons, and there is quite a difference between ns2 and the real stacks. I’m yet unsure if this result of analysis is to do with the way I’ve looked at the data so far, rather than what the data really means. I’ll have to keep at it a bit.

Currently trying to use the listings package, but for some reason the code I’m putting in the lstlisting environment isn’t coming up. If I tell the environment to have a frame, I do end up with a frame in my document, just it is rather small and doesn’t have anything inside.

No idea what is up at the moment. Maybe I should attack this some more another time.

Really need to make a test case and see if I can reproduce it in a very simple latex file. If not, start adding stuff that is in my thesis (other packages and the like).

Update: It was the “draft” mode of the report document class that was doing this. No idea why.

Look over my 420 to see how I approached the implementation section.

Get other theses/documents so I can refer to these at times. Want to figure out how to go about my literature review last section.

Analyse sim-33 data.

Figure out a nice way to do listings of source code in my thesis.

Discuss with Tony where my previous work (420 report) fits in.

So I fixed up the simulation script for sim-33 to output all of its parameters and added random seed and made the get_parameters script work as it should. Then figured I needed to run at least 19000 simulations. Each of which takes a bit of time.

Connected a couple of clients, the VERY rough ETA was 400 hours at that point.

I then spent quite a bit of time playing around to first get everything together I needed to run my simulations on lab machines then to automate starting a simulation client on a lab machine.

A couple of hours later I had 100 machines simulating.

I’ve left them overnight and the simulation is now all but finished.

Nice to see my simulation framework software work so well! Seems to have scaled alright, too, I had never really imagined I’d find 100 machines to connect to the server.

Been working on updating my literature review section somewhat. Also did a little development work this morning.

So I’ve been attacking it with a new design and I’m up to sorting out my OPNET info a bit more to see how it fits in. The TCP information on OPNET’s site lists all sorts of features implemented. E.g. RFCs RFC-793 (Transmission Control Protocol), RFC-1122 (Requirements for Internet Hosts — Communication Layers), RFC-1323 (TCP Extensions for High Performance), RFC-2018 (TCP Selective Acknowledgement Options), RFC-2414 (Increasing TCP’s Initial Window), RFC-2581 (TCP Congestion Control), RFC-2582 (The NewReno Modification to TCP’s Fast Recovery Algorithm), RFC-3168 (The Addition of Explicit Congestion Notification (ECN) to IP). And two textbooks, TCP/IP Illustrated Volume I – Stevens, TCP/IP Illustrated Volume II – Wright/Stevens.

Sounds fairly complete!

Even seems to simulate a receivers advertised window and a persist timer. Huh. Haven’t noticed any other simulators to do this just yet.

Now I just need to know if any of this has been tested and validated!

New feature in simcd.py added just now at the request of Perry (really something I’ve been meaning to add for ages, but keep forgetting to).

sam@carceri ~/900/simulations/bin$ ./simcd.py --help
./simcd.py -- simulation client
 -s,--server  Specify simulation server (localhost)
 -p,--port    Port number on server (8195)
 -n,--name    Client name (hostname-client-pid)
 -t,--test    Test simulation specified by the argument
 -i           Test simulation number specified by the argument (only with -t)
 -c,--condor  Use condor cluster management software to submit simulation jobs
 -e,--ns-exe  Executable to use (ns)
 -d,--daemonise Daemonise after startup (go into background)
 -v           Verbose/debug information
 -N,--nice    Run simulation processes niced
 -h,--help    This usage statement

No, not the help message. The line that goes: -N,--nice Run simulation processes niced

Spent the bits of today that were not organising things doing practical work.

Wrote some python code to generate graphs of data in a CSV file. This allows me to quickly generate lots of graphs of simulation data output. Works pretty well, though I might add some more smarts later on if I see the need.

Then went about generating some data, which involved fixing things up so I can run simulations on many computers. Hesiod (laptop) and voodoo worked a charm, required a bit of work to get going on gentoo (desktop) again.

Curst (FreeBSD) was the biggest hassle. I am getting non-deterministic errors from the globaliser. Core dumps, illegal instructions, all sorts. On the same input it will work 9 times, then fail once or so. Crazy stuff. So erm, I just kind of did this to build the project:

[sam@curst:~/uni/900/nsc]$ while ! scons; do echo "again..."; done

Yeah. Huh. Anyway… That mostly worked, it looked like it got stuck trying to link Linux, the following error results:

g++-3.3 -Wl,-O1 -Wl,-Tlinux-2.4/linker-script-full-2.ld -shared -o linux-2.4/liblinux24.so [... lots of .o files ...]
/usr/lib/libm.so: could not read symbols: Invalid operation

Haven’t looked into why this is happening just yet.

TODO: add the ability to ‘nice’ the simulation so it doesn’t hose voodoo.

The laptop seems to run simulations fast enough. This is encouraging.

I had lots of other weird errors about the place, carceri had empty files for net/socket.parsed.c for some reason, which broke the Linux shared library.

Then subversion mysteriously decided to stop working for my repositories on curst and carceri. I couldn’t update. I found that the svn+ssh url had an extra ‘:’ in it, like is used when you use scp. This broke things. I can only assume it didn’t use to break things, because I’ve been using those checked out copies and updating them for quite some time.

I did run a few simulations and get a bit of data. One thing that didn’t help this project were the following two problems:

1. Our Internet connection is pretty broken: RTTs vary from 40ms to 2200ms, with a mean somewhere in between.
2. Wireless is up and down like a… (fill in simile here yourself)

This meant subversion updates wouldn’t work sometimes, ssh-ing to voodoo was painful and ssh sessions would die, simulation clients would time out also because of the routes changing and no data getting through for so long.

So overall things can only get better, and I did make some useful headway.

I haven’t written anything for a couple of work days, so here we go.

Kept working a lot on my literature review chapter. It isn’t obvious how this should be set out. I went through and gave it some structure, but I am currently unhappy with it.

Talked to Tony, which was quite productive. Discussed the problems I’m having with the literature review chapter and how I might go about addressing them. Came up with a plan, I’ll implement this plan in the following days and see if a new structure will work. Don’t want to get this wrong, would be wasted writing time.

Spent more time doing practical work exploring the parameter space of my nimrod/o result. Now working on automating things a bit more so I can run simulations and analyse results faster, at the suggestion of Tony. Taking some time now to write some scripts and update my systems so I can run simulations and get results easily.

This is taking a little while. Especially since a) my wireless is down and b) our Internet connection is rather broken, latencies average about 900ms, range from 40ms to 2200ms with a standard deviation of 500ms or so. Things are pretty broken. I’d get a better connection on the other side of the world! (Though perhaps not with NTL in Cambridge, UK)

I looked into some of my simulation output just now. I had set up my simulations badly, all I varied were the stacks being simulated and the forward queue size.

What I found was that with linux (linux26 especially) cross traffic, throughput for the measured stream was low. With ns2 cross traffic, throughput for the measured stream was high. This sort of result seems to tell us that Linux is a fair amount more aggresive. I think I’ve tried to measure this in the past and failed. Maybe there is some critical mass of Linux TCP streams that shows up like this?

The simulation is stupid MAINLY because of the RTTs: they are all set at like 520ms or something, slightly different in both paths. I should change this a bit and play around with that.

Doing some more practical work: exploring parameter space of the nimrod/o result. My initial go at this came out with I simulation I conservatively estimated at taking 58 years. My next go is currently running.

I wrote on paper yesterday something I was thinking about:

Do we need to explain all the little things that could be simulated with TCP: the TCP protocol is standardised, but there are lots of possible differences… RFCs, from 793 onwards. They specify some bits are optional. (Also RFCs aren’t always perfect and leave some stuff up to interpretation). Also illustrate that implementations don’t always implement standards as they should; wether from bugs or by choice. eg. Windows in my measurement paper.

Simulating TCP accurately would simulate all this features, right?

Question: where does this spiel fit in the thesis then?

Now that I’ve finished playing with my laptop a bit, I’ve sat down and put in a hard days work.

Added notes on: SSFNet, SSF/DaSSF, ns-2, gtnets (not much), x-Sim, ATM-TN, NCTUns (stolen from previous paper for now).

Added lots of references for all the above and read through a bit of stuff to make the notes. Fixed up a few more things in my bibliography while I was at it.

Currently my literature review chapter is full of notes and not much structure design. Need to revisit it once I’ve finished filling it full of notes and rework it into something structured and see how well it works.

Added a wordcount rule to my makefile too.

No practical work today, must get back to some of that tomorrow.

Today I received my shiny new Inspiron 6000 laptop from Dell. It was preinstalled with Windows XP Home.

I chucked an Ubuntu Breezy RC1 CD in, resized my windows partition down with the installed, and let the install do its stuff.

A while later I’m using the laptop and I haven’t manually set up a thing. Everything so far just works.

• Xorg set up to use 1680×1050 widescreen, works fine.
• Sound works fine.
• Intel 2200 wireless works a charm.
• The Broadcom 10/100 NIC seems to be detected, though I haven’t bothered to use it just yet.
• Touchpad works fine in X.
• Speedstep seems to work, most time is spent in 600MHz mode, but when I do something like load The Gimp, it hops up to 1.6GHz.
• Battery monitor in gnome set up by default, this also works (no idea if its estimate of time remaining is correct just yet).
• Sound on/off, volume up/down special keys on the front of the laptop work.
• Plugging my digital camera in came up with a “would you like to download pictures from your camera” dialog box. This worked perfectly.
• Going into standby works perfectly.

I was expecting to spend the rest of the day (and them some) setting Linux up to work nicely on this laptop. Instead I haven’t spent a second setting it up.

I’m not happy with a default gnome setup, but that is a personal preference thing. I’ll play around a bit and see what I can get working that makes the laptop a bit nicer to work on.

Good work Ubuntu project! Linux feels very accessible.

From this blog I found a little Perl program to make it easy for me to cite RFCs. It requires downloading an xml index of RFCs from this page.

# wget http://www.educatedguesswork.org/xml2bibtex
# wget ftp://ftp.isi.edu/in-notes/rfc-index.xml
# chmod +x ./xml2bibtex
# ./xml2bibtex rfc-index.xml 

That would work if I had the Perl modules installed. I can’t be bothered with this and don’t have admin on the machine, so I just wrote my own version of xml2bibtex in Python instead.

Running this,

# python rfc-xml2bibtex.py rfc-index.xml | less
\@misc{RFC0001,
    author = {S. Crocker},
    title = {{Host Software}},
    howpublished = {RFC0001},
    month = {April},
    year = {1969}
}


\@misc{RFC0002,
    author = {B. Duvall},
    title = {{Host software}},
    howpublished = {RFC0002},
    month = {April},
    year = {1969}
}

[snip]

Doing some research on OMNeT++ at the moment. There is apparently a paper on it published in the European Simulation Multiconference (ESM’2001). After spending quite some time searching for it, I have given up. Maybe it isn’t published electronically? Seems silly to me if that is so.

Just reading about the TCP models in OMNeT++ now. Would really like to know how validated they are. Further reading and research required…

Thanks go to Ian McDonald, who pointed me to Remus.

Quote from the Remus page:

This is remus (Router EMUlation Services is the backronym), a small set of hacks on top of the Linux policy routing engine to allow a single server to emulate part or all of a network topology.

Not sure if I’ll use it, but it is an interesting little project.

I’ve investigated my perceived differences with object request time and FullTcp some more. The results are not very good or interesting. Seems that things are mostly fairly similar, I produced some very boring graphs. Just looking at the times for the first 3 objects in the connection. Hmm.

Update: it seems like the difference comes out after the first 3 object requests. Hmm. Looking into that further now.

Update 2: looks like another situation where ns2 falls somewhere roughly between freebsd and linux. FreeBSD seems better in this situation for some reason…

I have sorted out my background chapter somewhat. I’ve written a lot of brief notes under each section and subsection and subsubsection. Below is the current structure of it:

--------------------
Background
 . . Discrete event simulation
 . . . . Anatomy of a discrete event simulator
 . . . . . . Discrete events
 . . . . . . Global event list
 . . . . . . Simulated time
 . . . . . . Simulation process
 . . . . . . Serial versus parallel simulation
 . . . . . . Simulation components
 . . . . . . Models and abstraction
 . . . . Simulating a computer network
 . . . . . . Nodes
 . . . . . . Links
 . . . . . . Agents
 . . . . . . Layered protocols
 . . . . . . Non-blocking protocols
 . . . . . . An simple scenario
 . . Real world code in simulation
 . . . . Cradle code
 . . . . . . A simple cradle example: lwIP
 . . . . . . A more complex cradle example: BSD network stack
 . . . . . . Simulator interactions
 . . . . Multiple instances: the re-entrancy problem
 . . . . Kernel code in user space
 . . . . . . Alpine
 . . . . . . ENTRAPID
 . . . . . . FreeBSD network stack extensions
 . . . . . . User Mode Linux
 . . Previous work

Now, the section I’m most hesitent about is “Previous work”. I should talk to Tony about how this should be addressed. My guess is a reasonable explanation of what it entailed and its results, then cite it. Not sure if it should have its own section or how long this will be? Certainly worth thinking about.

My “real world code in simulation” section really turned into something quite network simulation centric. Not the end of the world, but I envisioned it somewhat more generic. Might require an explanation saying “this section gives background on real world code in simulation with examples in the area of network simulation”.