Libtrace 3.0.17 has finally been released.

This release adds some new convenience functions to the libtrace API and fixes a number of bugs, many of which have been reported by users.

The major changes in this release are:
* Added API functions for getting the IP address from a packet as a string.
* Added API functions for calculating packet checksums at the IP and transport layers.
* Fixed major bug where the event API was not working with int: inputs.
* Fixed broken checksum calculations in tracereplay.
* Fixed bug where IP headers embedded inside ICMP messages were not being anonymised by traceanon.
* Added API support for working with ICMPv6 headers.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

Exporting from NNTSC is now back to a functional state and the whole event detection chain is back online. Added table and view descriptions for more complicated AMP tests; traceroute, http2 and udpstream are now all present. Hopefully we can get new AMP collecting and reporting data for these tests soon so we can test whether it actually works!

Had some user-sourced libtrace patches come in, so spent a bit of time integrating these into the source tree and testing the results. One simply cleans up the libpacketdump install directory to not create as many useless or unused files (e.g. static libraries and versioned library symlinks). The other adds support for the OpenBSD loopback DLT, which is actually a real nuisance because OpenBSD isn't entirely consistent with other OS's as to the values of some DLTs.

Helped Nathan with some TCP issues that Lightwire were seeing on a link. Was nice to have an excuse to bust out tcptrace again...

Looks like my L7 Filter paper is going to be rejected. Started thinking about ways in which it can be reworked to be more palatable, maybe present it as a comparative evaluation of open-source traffic classifiers instead.

Added a data parser module to NNTSC to process the tunnel user count data that we got from Lightwire. Managed to get the data going all the way through to the event detection program which spat out a ton of events. Spent a bit of time combing through them manually to see whether the reported events were actually worth reporting -- in a lot of cases they weren't, so I've refined the old Plateau and Mode algorithms a bit to hopefully resolve the issues. I also employed the Plunge detector on all time series types, rather than just libprotoident data, and this was useful in reporting the most interesting behaviours in the tunnel user data (i.e. all the users disappearing).

Added a couple of new features to the libtrace API. The first was the ability to ask libtrace to give you the source or destination IP address as a string. This is quite handy because normally processing IP addresses in libtrace involves messing around with sockaddrs which is not particularly n00b-friendly. The second API feature was the ability to ask libtrace to calculate the checksum at either layer 3 or 4 based on the current packet contents. This was already done (poorly) inside the tracereplay tool, but is now part of the libtrace API. This is quite useful for checksum validation or if you've modified the packet somehow (e.g. modified the IP addresses) and want to recalculate the checksum to match.

Also spent a decent bit of time reading over chapters from Meenakshee's report and offering plenty of constructive criticism.

Just a lonely two day week while everyone else was still on holiday.

Released a new version of libtrace (3.0.16) - now Richard's ring buffer code is out amongst the wide world and hopefully our users won't find too many bugs in it.

Got back into writing my paper on L7 Filter. Most of the content is there now, although I'm not entirely convinced that the way I have structured the paper is quite right. It's much more readable the way I have it now, but it looks more like a bulleted list than a technical paper.

Meenakshee's LPI collector worked pretty well running on some trace files over the break, which was pleasing. Next step is to get it working on our newly functional ISP capture point. Tested the capture point out by running some captures over the weekend - aside from a bug in the direction tagging everything looks good, so we have at least one working capture point.

Libtrace:
Managed to get native BPF socket capture exporting correctly over the RT protocol. Changed the build system to make it possible to export captures taken using a native socket interface over RT to a machine running a different OS to the capture host, e.g. capture using Linux Native, export to a FreeBSD box.

WDCap:
WDCap now builds and runs on both Mac OS X and FreeBSD. Also changed the way the disk output module names files, based on some code submitted by Alistair King. You now specify your output filename format using strftime-style conversion modifiers, which offers a bit more flexibility to users rather than them being stuck with our particular file naming convention.

lpi_collector:
Continued working closely with Meenakshee on the new collector. Designed a binary format for exporting our collector messages called the libprotoident collector protocol (or LPICP for short).

L7 Filter:
Finished collecting traces for most of the protocols I wanted to test with L7 Filter and collated the initial results. Wrote a blog post about it (https://secure.wand.net.nz/content/case-against-l7-filter) and started working on a paper.

Short week this week - took leave on Thursday and Friday.

Released a new version of libtrace (3.0.15) on Monday. Mostly just a few little bug and build fixes, but it had been a while since the last release. Also submitted a patch for the FreeBSD libtrace port which had been broken for a very long time.

Did a bit more refinement on my Plunge and ArimaShewhart event detectors. They're at a stage now where the number of false positives is close to none. False negatives are a bit harder to identify, of course. The next sensible step is probably to think about testing against real-time data and manually validate the events as they roll in.

Spent a day looking at the latest LPI data from a live analysis I have running on our ISP monitor. Managed to get some up-to-date stats on application usage for last September but haven't had a chance to look over it in detail yet.

I did note a bit of an increase in the amount of unknown UDP traffic, so chased up a few of the more common patterns. Have added 3 new protocols to libprotoident as a result: ZeroAccess (a trojan), VXWorks Exploit and Apple's Facetime / iMessage setup protocol.

Added a new anomaly detector to our network event monitor: the Plunge Detector. The basic aim is to detect situations where an otherwise active time series plunges to a very low (or zero) value. Sounds simple, but kinda tricky to do in a generic fashion. The general algorithm is track the median and minimum observed values over the past N measurements and then raise an alarm when the current value is both significantly below the median and the minimum observed values.

Spent much of the week testing both the new Plunge detector and the Shewhart detector against the various LPI time series in my test data set. Lots of refinement going on with both detectors, but starting to get pretty happy with the results.

Started working towards a new libtrace release - mostly just a few little bug fixes and tidyups. Part of the release process is to test it on a FreeBSD machine, but the old emulation image doesn't work with the new emulation network. Set up a FreeBSD 9 machine so that Brendon could make a new image, which was a lot more painful than it should have been. Managed to get libtrace tested and passed the machine over to Brendon for imaging - I expect a decent rant in his weekly report about that step of the process to :)

Our paper on libtrace entitled "Libtrace: A Packet Capture and Analysis Library" has been officially published in this month's edition of ACM Computer Communication Review.

It has been a bit of a battle over the years to find a venue that was willing to publish a paper on libtrace, as the direct scientific contribution of libtrace itself is subtle. It was also difficult to articulate exactly how libtrace is so much easier and pleasant to work with compared to other trace analysis libraries. Often the improvements present in libtrace were dismissed out of hand as being nice but not necessary.

For example, capture format agnosticism was dismissed by some reviewers as mostly pointless because they never needed to work with a trace format other than pcap. The performance enhancements were similarly discredited because it was just easier to "buy a faster CPU" or because you could just use a separate zcat process to decompress the trace instead (hence the explicit discussion of the difference between using a separate process + pipe versus the threaded approach employed by libtrace).

As a result, we often had to go back to the drawing board and think more carefully about how to "sell" each of the enhancements in libtrace and clearly explain the reasoning behind each design decision. Eventually we managed to find the right combination of venue and tone that allowed us to finally get a submission accepted.
Hopefully this will lead to more network researchers learning about libtrace and adopting it for use in their own research and analysis tasks.

A copy of the paper can be downloaded from here.

Released libtrace 3.0.14 - mostly just a bug fix release. I also separated the I/O stuff into a separate library so that it can be used outside of libtrace.

Took a quick look at maji again to see if we can use it as part of the MSI project. Fixed up some bugs that became apparent when exporting lots of flow records. Also decided that maji would work a lot better if it underwent a major design change, but resisted the temptation to do so for now.

Secured the RT exporter connected to the live capture point so that only WAND machines can connect to it - someone from a lightwire address had connected to it and sent something invalid which broke the whole wdcap process. The RT exporter also now handles invalid client responses better :)

Started looking at Andreas' time series anomaly detection code. The existing system only really works with offline data, so the first goal is to get it running against a "live" input source.