Sam’s Network Simulation Cradle Blog

After adding SCTP decoding functionality to Perry’s tracedump I’ve found that SCTP transmit INIT (synonymous with SYN) packets.

There is no reply. For some reason the receiving host can’t find a route or something. At first, the sending host seemed to fail to find a route, then I added sctp_globals to the globals file and I got the INIT packets.

Are ugly! Check it out:

#0  sctp_sf_do_5_1B_init (ep=0x8642130, asoc=0x0, type=
      {chunk = SCTP_CID_INIT, timeout = SCTP_EVENT_TIMEOUT_T1_COOKIE, other =
1, primitive = SCTP_PRIMITIVE_SHUTDOWN}, arg=0x8650930, commands=0xbfffd988)
    at net/sctp/sm_statefuns.c:203
#1  0x40f66805 in sctp_do_sm (event_type=SCTP_EVENT_T_CHUNK, subtype=
      {chunk = SCTP_CID_INIT, timeout = SCTP_EVENT_TIMEOUT_T1_COOKIE, other =
1, primitive = SCTP_PRIMITIVE_SHUTDOWN}, state=SCTP_STATE_CLOSED,
    ep=0x8642130, asoc=0x0, event_arg=0x8650930, gfp=32)
    at net/sctp/sm_sideeffect.c:806
#2  0x40f5b56d in sctp_endpoint_bh_rcv (ep=0x8642130)
    at net/sctp/endpointola.c:377
#3  0x40f736a5 in sctp_inq_push (q=0x864214c, packet=0x8650930)
    at net/sctp/inqueue.c:112
#4  0x40f69bdd in sctp_backlog_rcv (sk=0x8641d30, skb=0x8650930)
    at net/sctp/input.c:259
#5  0x40f69af0 in sctp_rcv (skb=0x863e698) at net/sctp/input.c:213
#6  0x40ed7468 in ip_local_deliver_finish (skb=0x863e698)
    at net/ipv4/ip_input.c:259
#7  0x40ed6e0a in ip_local_deliver (skb=0x863e698) at net/ipv4/ip_input.c:299
#8  0x40ed7674 in ip_rcv_finish (skb=0x863e698) at net/ipv4/ip_input.c:364
#9  0x40ed6f88 in ip_rcv (skb=0x863e698, dev=0x863f890, pt=0x412e2454)
    at net/ipv4/ip_input.c:434
#10 0x40eb8500 in netif_receive_skb (skb=0x863e698) at net/core/dev.c:1530
(More stack frames follow...)

Or better yet:

#0  sctp_add_cmd_sf (seq=0xbfffdc98, verb=SCTP_CMD_TIMER_RESTART, obj=
      {i32 = 2, u32 = 2, u16 = 2, u8 = 2 '\002', error = 2, state = SCTP_STATE_COOKIE_WAIT, to =
SCTP_EVENT_TIMEOUT_T1_INIT, counter = 2, ptr = 0x2, chunk = 0x2, asoc = 0x2, transport = 
0x2, bp = 0x2, init = 0x2, ulpevent = 0x2, packet = 0x2, sackh = 0x2}) at include/net/sctp/sm.h:439
#1  0x40f671b9 in sctp_cmd_interpreter (event_type=SCTP_EVENT_T_TIMEOUT,
    subtype=
      {chunk = SCTP_CID_INIT_ACK, timeout = SCTP_EVENT_TIMEOUT_T1_INIT, other = 2, 
primitive = SCTP_PRIMITIVE_ABORT}, state=SCTP_STATE_COOKIE_WAIT,
    ep=0x8628fe8, asoc=0x864d220, event_arg=0x2,
    status=SCTP_DISPOSITION_CONSUME, commands=0xbfffdc98, gfp=32)
    at net/sctp/sm_sideeffect.c:1251
#2  0x40f668a1 in sctp_side_effects (event_type=SCTP_EVENT_T_TIMEOUT,
    subtype=
      {chunk = SCTP_CID_INIT_ACK, timeout = SCTP_EVENT_TIMEOUT_T1_INIT, other
= 2, primitive = SCTP_PRIMITIVE_ABORT}, state=SCTP_STATE_COOKIE_WAIT,
    ep=0x8628fe8, asoc=0x864d220, event_arg=0x2,
    status=SCTP_DISPOSITION_CONSUME, commands=0xbfffdc98, gfp=32)
    at net/sctp/sm_sideeffect.c:840
#3  0x40f6682e in sctp_do_sm (event_type=SCTP_EVENT_T_TIMEOUT, subtype=
      {chunk = SCTP_CID_INIT_ACK, timeout = SCTP_EVENT_TIMEOUT_T1_INIT, other
= 2, primitive = SCTP_PRIMITIVE_ABORT}, state=SCTP_STATE_COOKIE_WAIT,
    ep=0x8628fe8, asoc=0x864d220, event_arg=0x2, gfp=32)
---Type <return> to continue, or q <return> to quit---
    at net/sctp/sm_sideeffect.c:809
#4  0x40f65a6a in sctp_generate_timeout_event (asoc=0x864d220,
    timeout_type=SCTP_EVENT_TIMEOUT_T1_INIT) at net/sctp/sm_sideeffect.c:266
#5  0x40f65b30 in sctp_generate_t1_init_event (data=140825120)
    at net/sctp/sm_sideeffect.c:288
#6  0x40f8dd33 in run_timer_list () at kernel/timer.c:321
#7  0x40f8d0c3 in timer_bh () at kernel/timer.c:696
#8  0x40f887ad in nsc_timer_interrupt () at nsc/support.c:63
#9  0x40f9277e in LinuxStack::timer_interrupt (this=0x85c2568)
    at nsc/sim_support.cpp:296
(More stack frames follow...)

ACM Transactions on Modelling and Simulation looks interesting. Seems to have published a few papers on network simulation stuff before.

SCTP seems to be initialised now. Changing the order in which the object files were linked got the initialisation functions called right. I wouldn’t exactly call this an elegant solution.

After reading some more about SCTP I figured it would be worth spending an afternoon on.

Looks like I’ve basically got it to the stage where it should just initialise and work. However, instead it segfaults.

I tracked this down to the fact that sctp_init is called before inet_init, which is a bad thing. Now, as far as I know, this should work: I’m just attempting to compile SCTP statically into the kernel.

But maybe everybody just uses it as a module. Perhaps I should add better module handling into NSC. Either that, or figure out how I can get SCTP to initialise itself AFTER inet. FreeBSD’s initialisation stuff is seeming much better right now.

Linux 2.4.27 is go. The upgrade path wasn’t too bad, but pretty much involved putting my changes to the few headers in asm/ back into the Linux 2.4.27 versions.

Changed file summary: * include/asm-i386/smp.h (smp_processor_id()) * include/linux/sched.h (capable()) * include/asm-i386/system.h (local_irq_*() macros) * include/asm-i386/softirq.h (local_bh_enable(), local_bh_disable(), __local_bh_enable() macros) * include/asm-i386/spinlock.h (spin, read/write locks) * kernel/softirq.c (tasklet_kill, __initcall removal) * include/asm-i386/processor.h (prefetch)

The biggest problem I know of with the parser at the moment is that it doesn’t parse out asm() statements. So if I have:

__asm__ __volatile__ (    "cmpl $0, -8(%0);"    
"jnz 2f;"   
"1:;"   
".subsection 1nt"    
""     
".ifndef "
 ".text.lock." 
"clnt"    
"nt" 
".text.lock." 
"clnt"   
 ":nt"   
".endifnt"  
"2: pushl %%eax; pushl %%ecx; pushl %%edx;" 
"call %c1;" 
"popl %%edx; popl %%ecx; popl %%eax;"   
"jmp 1b;"   
".previousnt"   
: : "r" (ptr), "i" (do_softirq)     );

The ‘ptr’ and ‘do_softirq’ bits will not be “globalised”. Would be nice to have this functionality, though it isn’t of the utmost importance.

Working on upgrading simulation to Linux 2.4.27 now. Thought I had done it last week but I’d made a bit of a mistake.

I did a 5% random loss test and compared simulation and emulation. There was actually less then 1% difference in the results reported. I’ll conclude this as a win and move on I think. I am really quite intrigued with SCTP at the moment… Maybe it isn’t worth pursuing just yet, though.

IBM is offering Thinkpads and stuff for some really difficult programming challenges: these can be found here.

One thing they noted was some stuff to do with SCTP. Perhaps it is time to have a play with SCTP.

The morning has been wasted with getting my desktop going with a new mainboard and ram. Is up and running now, though I have yet to get accelerated X going properly. And I need to compile a new module for the 1Gb nic. The 10/100 one works perfectly. Sound seems to work nicely as well. Maybe I should consider plugging my cdwriter in at some point as well.

I still plan to change some hardware in vegas as well: need to sort out RAM issues: check why 256MB sticks aren’t working and revert to the 192MB of RAM it had before if they are not going to work. Also want to swap 80GB hdds.

This is probably a mission for this evening.

For now I should do some work.

14:00 < @sammydre`_`> a) valgrind shows nothing
14:00 < @sammydre`_`> b) it happens on freebsd and not linux
14:00 < @sammydre`_`> c) the program runs on many other test cases fine
14:01 < @sammydre`_`> d) it happens inside an STL set. I tried changing from 
          hash_set to set, made no difference
14:01 < @sammydre`_`> e) the bit of code it crashes in is a find(), searching for 
          'csum`_`fold'. The exact same find had been performed a little earlier 
          and worked perfectly
14:02 < @sammydre`_`> f) it is reproducable
14:02 < @sammydre`_`> g) the set is not modified between the find() that fails and 
          the one that works (knowingly, at least)
14:05 < @sammydre`_`> j) I do compile with -Wall

Update:

I found the bug. It was an overflow of a static buffer. I had assumed an error like this would be picked up with valgrind, but I guess not. Very silly error. It only happened when a very specific token was too long. For now I’ve increased the buffer size tenfold. I’m not too sure how to solve this problem in a better way; it would seem I need to understand how flex allocates the buffer size of yytext. I tried a realloc at first, but that didn’t work. Anyway, it works now and Linux almost builds on FreeBSD.

The problem now is that it doesn’t link when I use the linker map (which is quite necessary for operation). It does link fine when I don’t use the linker map. This will need further investigation I guess. Almost there.

Finding bugs like this is never a fun thing. In this case I just iteratively narrowed down my input till I had a small amount of input that broke it, and a minor change to the input that made it work. Once I got it down to this point I could formulate theories on why it was breaking. I added some debug code to show what was happening in what turned out to be the erraneous function and I eventually noticed the small buffer size.

Attempting to get Linux working on FreeBSD again. I was getting random segfaults in the parser last time I tried. This is something I’ve wanted to work for ages and I haven’t had the time to pursue.

Played with getting Linux closer to a blocking application.

Changed waitqueue_active so it always returns 1. I did this because it would never return true otherwise because there is never an application to actually wait on a queue. This has the effect of making __wake_up get called; which now in turn calls nsc_wakeup which calls the simulator wakeup function. I’m not sure how much difference this makes just yet.

There is some difference in packet traces, but they are fairly close. Should perhaps check some higher level stuff and also RTO’s too see how different/close things are now. I’m happy enough that Linux is performing as it should, even if there are still some differences.

So it seems that all differences I’ve observed so far with Linux are created by the application (model). I’ve played a bit with tcpperf trying to make it act like what I’ve got in simulation and suddenly the traces are a lot more similar.

There are still differences. Mostly to begin with I get some small timing differences; which I don’t find particularly suprising.

So. I’m fairly certain that what differences there are (and it seems pretty bloody close now) are due to differences in the modeled application. I don’t think it is really worth pursuing this much further.

What I could do is work on making simulation work like a blocking application better; I did this for FreeBSD. However I don’t see this as particularly important.

Really need to figure out where to go next.

The goal for version 1.0:

Have a game someone can download or put on their computer and change some simple configuration: server to connect to, player name, player type and then play. Fight other people, get frags, respawn, hit frag limit, go to next level. Chat in game.

What we need to get done for this goal.

Networking

• Server should probably tell the client when they get hurt. And certainly when they die. DONE
• It should also have a level list that it goes through round robin. Levels change once a frag limit is hit. This should be scripted by some lua file. DONE

Content

• player models
• powerups
• guns
• gui graphics
• sounds

Graphics

• Visual feedback of hurting others/getting hurt.
• Drawing of weapon that you shoot with perhaps.
• Important stuff drawn on HUD. Try and minimise HUD. MOSTLY DONE

Gameplay

• Players need proper health and they should be able to die. DONE
• The server should be able to tell the player to respawn. DONE
• Feedback of when you get damaged is necessary.
• Some way of knowing who the other players are when you look at them is also necessary.
• Score list ingame. Perhaps showing ping would be nice on this. DONE

OS

Should work on Linux and Windows. Ideally we will be able to make optimised builds for both. Currently it works perfectly on both OS’s, I see no reason that this should change.

Gameplay stuff

Theme: Ninja, pirate, monkey, robot.

Each has different abilities.

The monkey

Flies about. Default weapon: throws some object, fairly short range. Like the current ‘rocket’ weapon. Other weapons? Already have monkey head model, this will be fine.

The robot

Walks about slowly. Perhaps has some laser/sniper rifle/rail gun weapon. This shouldn’t be too hard to implement. Need a model. Can’t be too hard to make simple robot model. It can be clunky. And perhaps not Bender.

The ninja and pirate sounds harder to do, so just a robot and monkey is probably a good starting point.

Powerups

Double damage? Tri-damage? Quad-damage? Something. Health.

Not all code follows this currently. Try to make any new code you write follow this document. Old stuff can be slowly migrated over. Don’t worry too much if your code doesn’t match this, but it would be nice if at some point in the not too distant future all the code followed these guidelines.

I think a good example of these can be found in entity_manager.cpp and entity_manager.h

Class names

Classes should begin with the letter C and then use upper case for the beginning of every world in the name. No underscores should be used.

Example: CWorld, CDisplayManager

Class member function names

All words in the name should begin with capitals like class names.

Example: CWorld::Draw(), CDisplayManager::DrawTriangleFans()

Class member variables

This is perhaps the most inconsistently named thing in BuNg at this time. The original coding style used m_ at the beginning of member variables and I believe this is the way to go. I wouldn’t worry too much about spending lots of time updating current code to this style; it should be slowly moved across.

Member variables (in fact, variables in general) should use lower case and underscores where appropriate.

Examples: CWold::m_particle_system, CDisplayManager::m_display

General/temporary variables

See above.

Global variables

I try to cut these down as much as possible. If you must use a global that needs to be accessible from multiple parts of BuNg it can be contained in CWorld.

Enums

I’m not sure how we should name enums. Thoughts?

{ style

This can best be illustrated with examples:

// Class 
class CWorld
{
    CWorld();
};

// Function
CWorld::CWorld()
{
    int i;
    // Code block inside function
    for(i = 0; i < 10; i++) {
        Log("%dn", i);
    }
}

Indentation and tabs

Tabs should be written to file as spaces. The indentation width used should be 4 spaces.

Ignore the title of this blog entry, it is Brendon’s fault.

At any rate, to summarise the progress (or lack thereof) of my simulation vs. emulaiton studies:

The first difference of notive is that packet 16 has the PUSH flag in simulation, not in emulation. Continued searching into this made me come up only with the answer that this does not matter. Just the difference in applications is generating this.

I have continually searched to make sure there is enough data written by the application; this seems to be the case in simulation and emulaiton.

Attempting to figure out why the 700-ish byte packet is sent has shown me there is a list of quite a few packets to send out. The first sk_buff is 700 or so bytes, but there are a whole lots of 1448 byte ones after this (1448 is the MSS).

Soooooooooooooooooo. Why is there an sk_buff with that sized packet in there, and why is a full sized packet not sent out? This is what is happening on the real test network, remember, not in simulation.

Emulation:

01 tcp_write_xmit: skb->len: 1024
02 tcp_write_xmit: skb->len: 1448
03 tcp_write_xmit: skb->len: 1448
04 tcp_write_xmit: skb->len: 1448
05 tcp_write_xmit: skb->len: 1448
06 tcp_write_xmit: skb->len: 1448
07 tcp_write_xmit: skb->len: 1448
08 tcp_write_xmit: skb->len: 1448
09 tcp_write_xmit: skb->len: 704
10 tcp_write_xmit: skb->len: 1448
11 tcp_write_xmit: skb->len: 1448
12 tcp_write_xmit: skb->len: 1448
13 tcp_write_xmit: skb->len: 1448
14 tcp_write_xmit: skb->len: 1448
15 tcp_write_xmit: skb->len: 1448
16 tcp_write_xmit: skb->len: 1448
17 tcp_write_xmit: skb->len: 1448
18 tcp_write_xmit: skb->len: 456
19 tcp_write_xmit: skb->len: 1448
20 tcp_write_xmit: skb->len: 1448

Simulation

printk[0]: 01 tcp_write_xmit: skb->len: 1024
printk[0]: 02 tcp_write_xmit: skb->len: 1448
printk[0]: 03 tcp_write_xmit: skb->len: 1448
printk[0]: 04 tcp_write_xmit: skb->len: 1448
printk[0]: 05 tcp_write_xmit: skb->len: 1448
printk[0]: 06 tcp_write_xmit: skb->len: 1448
printk[0]: 07 tcp_write_xmit: skb->len: 1448
printk[0]: 08 tcp_write_xmit: skb->len: 1448
printk[0]: 09 tcp_write_xmit: skb->len: 1448
printk[0]: 10 tcp_write_xmit: skb->len: 1448
printk[0]: 11 tcp_write_xmit: skb->len: 1448
printk[0]: 12 tcp_write_xmit: skb->len: 1448
printk[0]: 13 tcp_write_xmit: skb->len: 1448
printk[0]: 14 tcp_write_xmit: skb->len: 1448
printk[0]: 15 tcp_write_xmit: skb->len: 1024
printk[0]: 16 tcp_write_xmit: skb->len: 1448
printk[0]: 17 tcp_write_xmit: skb->len: 1448
printk[0]: 18 tcp_write_xmit: skb->len: 1448   
printk[0]: 19 tcp_write_xmit: skb->len: 1448
printk[0]: 20 tcp_write_xmit: skb->len: 1448   

I did check earlier that there was enough data to send. Or I was fairly certain that that was the case. This needs to be analysed further. Time to go home for now though.

Boy the Opteron is a fast machine. Things seem to run over 4 times as fast as the Athlon MP 1800.

Really need to configure the network stacks/kernels for athlon 64 to compile them probably. FreeBSD and Linux wont compile. Funnily enough OpenBSD does.

For some reason in my SConscript I had "parser/parser global_file.txt >output.c" (effectively) and this was failing. Changing it to have a space before the greater-than symbol make it work. NO idea why.