ticket summary component version milestone type owner status created _changetime _description _reporter 334 traceconvert: pcap->erf results in different (incorrect) output than dagconvert (Endace) tools defect salcock new 2012-11-20T05:03:16+13:00 2012-11-20T05:03:16+13:00 "Hi all, I've run into an issue with traceconvert in convert a pcap file to an erf file. The result is different from dagconvert (from Endace). The erf file from dagconvert works for my purposes but traceconvert does not. Traceconvert (same as tracesplit) seems to lose 4 bytes (see rlen). I like libtrace and would like to see this investigated. I'll provide anything you need. Thanks! - Emiel $ dagconvert --version dagconvert (DAG 4.2.2) $Revision: 12512 $ using: libpcap version 0.9.4 $ tracesplit -H | head -1 libtrace 3.0.15 $ dagconvert -Tpcap:erf -i ./capture10.pcap -o ./out-dc.erf $ du -b out-dc.erf 7180 out-dc.erf $ traceconvert pcapfile:./capture10.pcap erf:out-tc.erf.gz $ gunzip out-tc.erf.gz $ du -b out-tc.erf 6940 out-tc.erf $ dagbits -f ./out-dc.erf decode | head print 1: file offset 0x0 ts=0x50470f4ab4caf709 2012-09-05 08:37:30.706222000 UTC type: ERF Ethernet dserror=0 rxerror=0 trunc=0 vlen=1 iface=0 rlen=84 lctr=0 wlen=66 pad=0x00 offset=0x00 dst=00:00:0c:07:ac:6e src=00:19:99:8c:be:77 etype=0x0800 ip: version=4 headerwords=5 tos=0 length=48 ip: id=59597 flags=0x2 fragmentoffset=0 ip: ttl=128 protocol=6 checksum=0x0000 $ dagbits -f ./out-tc.erf decode | head print 1: file offset 0x0 ts=0x50470f4ab4caf709 2012-09-05 08:37:30.706222000 UTC type: ERF Ethernet dserror=0 rxerror=0 trunc=0 vlen=0 iface=1 rlen=80 lctr=0 wlen=66 pad=0xfe offset=0x60 dst=00:00:0c:07:ac:6e src=00:19:99:8c:be:77 etype=0x0800 ip: version=4 headerwords=5 tos=0 length=48 ip: id=59597 flags=0x2 fragmentoffset=0 ip: ttl=128 protocol=6 checksum=0x0000" gemiel@… 351 libtrace/libwandio tries to auto-detect compression on erf files, detects compression on non-compressed files, fails libtrace-library 3.0 defect salcock new 2013-05-14T09:40:21+12:00 2013-05-14T09:40:21+12:00 "First, thank you very much for a great library. Now the bug: Since ERF format does not have a header and the first few bytes are timestamp, detecting compression reliably is problematic. libtrace (noticed first using tracesplit to convert to pcap) attempts to auto-detect compression. This mostly works, but not always. Case in point was one ERF file I tried to convert to PCAP. tracesplit failed (quietly, this is another bug). I traced the problem to the auto-detection of compression. In particular, the first two bytes of the buffer satisfied the test (buffer[0] == '\037' && buffer[1] == '\235'). Compressing the file first before passing to tracesplit fixed the issue. Here's a small proposed patch (I don't know if this is the right place to put it, but there does not seem to be another option: diff -aur libtrace-3.0.17/lib/format_erf.c libtrace-3.0.17.new/lib/format_erf.c --- libtrace-3.0.17/lib/format_erf.c 2013-03-05 16:28:51.000000000 -0800 +++ libtrace-3.0.17.new/lib/format_erf.c 2013-05-13 14:14:03.462456692 -0700 @@ -307,7 +307,7 @@ if (DATA(libtrace)->seek.exists==INDEX_UNKNOWN) { char buffer[PATH_MAX]; snprintf(buffer,sizeof(buffer),""%s.idx"",libtrace->uridata); - DATA(libtrace)->seek.index=wandio_create(buffer); + DATA(libtrace)->seek.index=wandio_create(buffer, 0); if (DATA(libtrace)->seek.index) { DATA(libtrace)->seek.exists=INDEX_EXISTS; } diff -aur libtrace-3.0.17/lib/format_helper.c libtrace-3.0.17.new/lib/format_helper.c --- libtrace-3.0.17/lib/format_helper.c 2013-03-05 16:28:51.000000000 -0800 +++ libtrace-3.0.17.new/lib/format_helper.c 2013-05-13 14:10:10.339441279 -0700 @@ -246,7 +246,8 @@ /* Open a file for reading using the new Libtrace IO system */ io_t *trace_open_file(libtrace_t *trace) { - io_t *io=wandio_create(trace->uridata); + int autodetect = (trace->format->type != TRACE_FORMAT_ERF); + io_t *io=wandio_create(trace->uridata, autodetect); if (!io) { if (errno != 0) { trace_set_err(trace,errno,""Unable to open %s"",trace->uridata); diff -aur libtrace-3.0.17/lib/trace.c libtrace-3.0.17.new/lib/trace.c --- libtrace-3.0.17/lib/trace.c 2013-03-05 16:28:51.000000000 -0800 +++ libtrace-3.0.17.new/lib/trace.c 2013-05-13 14:13:06.868724525 -0700 @@ -181,7 +181,8 @@ } } - libtrace->io = wandio_create(filename); + int autodetect = 1; + libtrace->io = wandio_create(filename, autodetect); if (!libtrace->io) return; diff -aur libtrace-3.0.17/libwandio/wandio.c libtrace-3.0.17.new/libwandio/wandio.c --- libtrace-3.0.17/libwandio/wandio.c 2012-05-27 19:07:07.000000000 -0700 +++ libtrace-3.0.17.new/libwandio/wandio.c 2013-05-13 14:04:54.814362447 -0700 @@ -124,7 +124,7 @@ #define DEBUG_PIPELINE(x) #endif -DLLEXPORT io_t *wandio_create(const char *filename) +DLLEXPORT io_t *wandio_create(const char *filename, int autodetect) { parse_env(); @@ -140,38 +140,39 @@ if (!io) return NULL; len = wandio_peek(io, buffer, sizeof(buffer)); - /* Auto detect gzip compressed data */ - if (len>=2 && buffer[0] == '\037' && buffer[1] == '\213') { + if (autodetect) { + /* Auto detect gzip compressed data */ + if (len>=2 && buffer[0] == '\037' && buffer[1] == '\213') { #if HAVE_LIBZ - DEBUG_PIPELINE(""zlib""); - io = zlib_open(io); + DEBUG_PIPELINE(""zlib""); + io = zlib_open(io); #else - fprintf(stderr, ""File %s is gzip compressed but libtrace has not been built with zlib support!\n"", filename); - return NULL; + fprintf(stderr, ""File %s is gzip compressed but libtrace has not been built with zlib support!\n"", filename); + return NULL; #endif - } - /* Auto detect compress(1) compressed data (gzip can read this) */ - if (len>=2 && buffer[0] == '\037' && buffer[1] == '\235') { + } + /* Auto detect compress(1) compressed data (gzip can read this) */ + if (len>=2 && buffer[0] == '\037' && buffer[1] == '\235') { #if HAVE_LIBZ - DEBUG_PIPELINE(""zlib""); - io = zlib_open(io); + DEBUG_PIPELINE(""zlib""); + io = zlib_open(io); #else - fprintf(stderr, ""File %s is compress(1) compressed but libtrace has not been built with zlib support!\n"", filename); - return NULL; + fprintf(stderr, ""File %s is compress(1) compressed but libtrace has not been built with zlib support!\n"", filename); + return NULL; #endif - } + } - /* Auto detect bzip compressed data */ - if (len>=3 && buffer[0] == 'B' && buffer[1] == 'Z' && buffer[2] == 'h') { + /* Auto detect bzip compressed data */ + if (len>=3 && buffer[0] == 'B' && buffer[1] == 'Z' && buffer[2] == 'h') { #if HAVE_LIBBZ2 - DEBUG_PIPELINE(""bzip""); - io = bz_open(io); + DEBUG_PIPELINE(""bzip""); + io = bz_open(io); #else - fprintf(stderr, ""File %s is bzip compressed but libtrace has not been built with bzip2 support!\n"", filename); - return NULL; + fprintf(stderr, ""File %s is bzip compressed but libtrace has not been built with bzip2 support!\n"", filename); + return NULL; #endif + } } - /* Now open a threaded, peekable reader using the appropriate module * to read the data */ diff -aur libtrace-3.0.17/libwandio/wandio.h libtrace-3.0.17.new/libwandio/wandio.h --- libtrace-3.0.17/libwandio/wandio.h 2012-03-13 19:00:22.000000000 -0700 +++ libtrace-3.0.17.new/libwandio/wandio.h 2013-05-13 14:12:02.611893323 -0700 @@ -236,7 +236,7 @@ * given file (if any), provided that libtrace was built with the appropriate * libraries. */ -io_t *wandio_create(const char *filename); +io_t *wandio_create(const char *filename, int autodetect); /** Returns the current offset of the read pointer for a libtrace IO reader. * " yuri@… 281 Add support for NetMon libtrace-library libtrace3 defect salcock new 2010-10-07T10:48:36+13:00 2010-10-07T10:48:36+13:00 "Starting to see a few traces using Microsoft NetMon, such as Wireshark sample traces (http://wiki.wireshark.org/SampleCaptures). It would be nice if libtrace could also read these traces. Some documentation of the format can be found at http://www.scritube.com/limba/engleza/computers/Netmon-Capture-File-Format14459.php The format doesn't seem too bad - just need to make sure we don't treat the frame table as actual frames and will need a TRACE_TYPE to TRACE_NETMON conversion and vice versa." salcock 275 libpacketdump does not parse IP options libtrace-library defect perry new 2010-07-15T15:16:46+12:00 2010-07-15T15:16:46+12:00 We can parse and display the values for TCP options, but not IP options. To be fair, we haven't exactly encountered many, but we should at least make an effort! salcock 242 Verify (then document) libtrace's threadsafety documentation enhancement perry new 2007-03-08T14:19:48+13:00 2007-03-08T14:19:48+13:00 "libtrace should be threadsafe as long as: * A trace object is only accessed from one thread at a time * Packets are only modified in one thread, or if it's unmodified, then it can be used by multiple threads. This should be testing to verify the assumption is correct, then when (successfully!) tested integrated into the wiki and doxygen documentation." perry 249 Implement attributes from Dean Pemberton's pcapstats program in tracereport(1) tools enhancement perry new 2007-04-21T15:46:13+12:00 2007-04-25T23:58:51+12:00 "List of attributes: * Total packet count (done) * IP packet count (done) * Other ethertype count (done I think) * TCP packet count (Done) * Empty TCP packet count * UDP Packet count (Done) * Empty UDP packet count * ICMP Packet count (done) * Empty ICMP Packet Count * IP Other (done) * Interpacket arrival time stats n/sum x/sum x**2/eean/varience/stddev/max * IP version report * IP header length histogram/n/sum x/sum x**2/mean/varience/stddev/max * IP flag combination report * IP offset combination report * IP ToS report * IP Length: histogram/n/sum x/sum x**2/mean/varience/stddev/max * IP TTL histogram/n/sum x/sum x**2/mean/varience/stddev/max * IP Proto (done) * Unique source IP's * Unique dest IP's * Unique UDP Source ports * Unique UDP Dest ports * UDP size (histogram/x/..../stddev/max) * " perry 250 suggestion for naming the files that tracereport generated tools enhancement perry new 2007-04-28T23:16:25+12:00 2007-04-28T23:16:25+12:00 For those .rpt files tracereport generates, they are making the directory looking very messy. To add the name of input tracefile in front of names of output files may make them look tidier. yww4 239 int: format can't read from openvpn tunnels libtrace-library 3.0 libtrace3 defect perry new 2006-12-20T17:44:31+13:00 2008-05-05T13:30:58+12:00 "When using the int: format you cannot read from openvpn tunnels. pcapint: works fine (reads as LINUX_SLL. tracedump int:openvpntunnel Wed Dec 20 17:40:39 2006 unknown Linux ARPHRD type 0xfffe Don't understand link layer type -1 in trace_get_payload_from_link() Unknown Protocol: 65535 45 b8 00 2d 00 00 40 00 3f 11 b8 a1 c0 a8 01 0c E..-..@.?....... c0 a8 00 0a 09 fc 09 fc 00 19 8e 56 07 00 10 00 ...........V.... c0 a8 00 0a 02 01 01 53 00 00 00 00 00 .......S..... tracedump -c 1 pcapint:openvpntunnel Wed Dec 20 17:42:43 2006 Linux SLL: Packet Type = OUTGOING Linux SLL: Hardware Address Type = 0xfffe Linux SLL: Hardware Address Length = 0 Linux SLL: Hardware Address = 00:00:00:00:00:00 Linux SLL: Protocol = 0x0800 " daniel@… 265 pcapint: does not set IF promiscuous on OpenBSD libtrace-library 3.0 libtrace3 defect perry new 2009-02-18T03:14:28+13:00 2009-02-18T03:14:28+13:00 Should not utilities such as tracepktdump set the capturing interface promiscuous on startup? On OpenBSD it does not. niclas.rosell@… 271 RT needs to allocate packet buffers more intelligently libtrace-library 3.0 libtrace3 defect salcock new 2010-01-22T11:25:54+13:00 2010-01-22T11:25:54+13:00 "Currently, we always allocate the maximum buffer size for each RT packet. Since I've had to increase this value to deal with jumbograms, I've started to get a bit concerned about the amount of memory we're wasting on each packet, given the relative rarity of such large packets (especially given most captures using RT will be snapping the packets first). What we should be doing is allocating a much smaller amount by default and only increasing it if we observe a packet large enough to require it. Also, we may still have to deal with RT records that are larger than the current maximum (due to some trickery involving fragmentation being done by the network card *after* some capture methods see the packet), so we may need to do something about that too." salcock 277 tracetop needs strict-aliasing fixes tools libtrace3 defect salcock new 2010-07-27T17:10:30+12:00 2010-07-27T17:10:30+12:00 "Without -fno-strict-aliasing, tracetop reports warnings when casting the sockaddr structures. We may want to fix this at some stage - for now I've just disabled strict aliasing in the Makefile.am. This should be an easier fix than the one for traceanon, at least." salcock 278 Libtrace doesn't pass many tests on OpenSolaris tests libtrace3 defect salcock new 2010-08-03T16:05:49+12:00 2010-08-03T16:05:49+12:00 "I've managed to make it build cleanly, but actual operation still fails miserably. Problems found so far: * Threaded IO doesn't work at all - disabling threading will enable you to read trace files. * Byte-ordering is wrong when reading DUCK traces - can't get the version number right. I suspect something is reporting the wrong byte order somewhere. * Event test hangs for a very long time * Wireless test hits an assert failure on line 90 Obviously it is going to take way too much effort to fix all these prior to 3.0.7, but I figured I should document them just in case someone actually cares about OpenSolaris in the future." salcock 30 tracesplit should be able to split by tcp connection tools 3.0 libtrace3 enhancement perry new 2006-07-10T10:47:52+12:00 2006-07-10T10:51:38+12:00 "I would like to be able to use the tracesplit utlity to split a large trace into a series of smaller traces where each smaller traces contains a single tcp connection. I imagine in the case of incomplete connections they could simply be thrown away when using this option. The UI might be something like {{{ tracesplit pcap:bigtrace.pcap -t smalltrace }}} " matt@… 232 Support input from iptables / ULOG libtrace-library 3.0 libtrace3 enhancement perry new 2006-10-26T16:35:51+13:00 2011-04-29T05:00:59+12:00 Add a new format that supports reading from IPtables ULOG targets. This would allow libtrace / wdcap to replace specter and other ULOG daemons. jamie 288 Investigate making directread and directwrite work libtrace-library libtrace3 enhancement salcock new 2010-11-30T14:45:14+13:00 2010-11-30T22:40:15+13:00 The current use of O_DIRECT seems to fail pretty badly, so I've disabled the directread and directwrite flags. I don't doubt that they could provide some performance improvements though, so it'd be neat if we could get them working properly. salcock 337 Add support for fragmentation to tracereplay tools libtrace3 enhancement salcock new 2013-01-07T14:21:05+13:00 2013-01-07T14:21:05+13:00 "At the moment, tracereplay just packs a sad if you try to replay a packet that is larger than the MTU of the interface you're trying to send it over. What would be really cool is if tracereplay could do its own IP fragmentation to fit the packet in the sender's MTU and send it that way. Will need to be pretty careful about packets in the source trace that were already fragmented, though..." salcock 255 Ensure DUCK code works properly with DAG 3.X drivers libtrace-library 3.0 libtrace3 task perry new 2007-09-05T16:33:01+12:00 2007-09-05T16:33:01+12:00 I've updated and tested format_dag_25 for regular DAG capture using the DAG 3 series of drivers, but I haven't taken the time to ensure recording DUCK information still works correctly with the newer software. We should definitely do this at some stage, but it remains low priority for now because, as far as I am aware, we're the only ones that use DUCK and we won't be using DAG 3 drivers on that capture point any time soon. spa1 262 Add support for changing compression level to tools tools enhancement salcock assigned 2008-05-05T09:39:46+12:00 2010-07-16T14:48:51+12:00 All of the libtrace tools that can create output files should have the ability to allow the user to change the output compression level. The default should also be something sensible, 6 or 1 would be obvious choices. jpc2 48 SWIG libtrace-library 3.0 libtrace3 enhancement perry assigned 2006-08-09T14:00:26+12:00 2006-09-17T18:42:54+12:00 We need swig bindings for python/ruby/perl anonymous