[Tool] tools/process/pcap/WiPal (v. 2010-01-13)

top

- Port WiPal to GCC 4.4.

 - wipal-list-frame: Enable using -f and -F to provide hexadecimal
   dumps of frames.

 - wipal-stats: The "STA counters" sections now include "total number
   of frames" and "average frames per microsecond" columns.  See
   WiPal manual.

 - Introduce wipal-plot-onoff to plot the "ON/OFF events" section
   reported by wipal-stats.  This requires a valid Python
   installation and Pycairo (http://cairographics.org/pycairo/).

 - WiPal library: every compound is now located inside the top-level
   "wpl" namespace.

 - Revamp the library's pcap input/output system.  This induces
   faster pcap operations (both for WiPal's binaries and the
   library).  But this is also a huge API change for those using the
   library.

   The most important changes are:
   * the pcapxx namespace becomes wpl::pcap,
   * pcapxx::descriptor<> becomes wpl::pcap::file<> (in
     wipal/pcap/stream.hh),
   * pcapxx::dumper becomes wpl::pcap::ostream (in
     wipal/pcap/ostream.hh),
   * introduction of a wpl::pkt namespace for file-format independent
     compounds.  To this regard, pcapxx::frame_descriptor<> becomes
     wpl::pkt::packet<> (in wipal/pcap/pkt.hh).  Its pcap_header()
     method becomes meta().
   * pcap files with indexes are now a separate class:
     wpl::pcap::indexed_file<>

 - Library: perform a lot of file and compound renamings.  Among the
   most important ones:
   * move everything from wipal/wifi/frame/filter to wipal/filter,
   * move everything else from wipal/wifi/frame to wipal/wifi.  The
     wifi::frame namespace does not exist anymore either.
   * filter::microseconds_stamper becomes filter::timetracker,
   * wifi::mactime_tracker becomes phy::timetracker.

 - Fix a few bugs, as usual. ;-)

@MISC{tools-process-pcap-WiPal-2010-01-13,
  author = {Thomas Claveirole and Marcelo Dias de Amorim},
  title = {{CRAWDAD} tool tools/process/pcap/WiPal (v. 2010-01-13)}, 
  howpublished = {Downloaded from http://crawdad.cs.dartmouth.edu/tools/process/pcap/WiPal},
  month = jan,  
  year = 2010
}
					

WiPal is a piece of software dedicated to IEEE 802.11 traces manipulation. 
It comes as a set of programs and a C++ library. A distinctive feature of WiPal 
is its merging tool, which enables merging multiple wireless traces into 
a unique global trace. This tool works offline on PCAP traces that do not need 
to be synchronized. WiPal also provides statistics extraction and anonymization 
tools, and its authors plan to extend it. WiPal’s key features are flexibility, 
ease of use, and efficiency.

Copyright © 2007, 2008, 2009 Université Pierre et Marie Curie — Paris 6

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation ; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.

Its maintainer is Thomas Claveirole (thomas.claveirole@lip6.fr). 
It is developped with the help of Marcelo Dias de Amorim (Marcelo.Amorim@lip6.fr).

Should you have any question, look at the Frequently Asked Questions web page
(http://wipal.lip6.fr/faq.html).

To build WiPal the following softwares are needed:

GCC.
The Boost C++ libraries. More specifically:
any,
array,
conversion/lexical_cast,
date_time,
filesystem,
foreach,
format,
multi_array,
optional,
preprocessor
smart_ptr,
string_algo,
tuple.
The GNU MP Bignum Library,
OpenSSL.

WiPal’s packaging follows the GNU conventions. An installation documentation is 
provided in the INSTALL file in the package’s root directory. However, with 
a standard system, the following commands should do the trick:

mkdir _build
cd _build
../configure
make
make install-strip
make check

On some systems, you might have to customize the configure script’s invocation. e.g.:

mkdir _build
cd _build
../configure CPPFLAGS=-I/foo/bar/libgmp
make
make install-strip
make check

WiPal's programs all use the same invocation scheme:

     wipal-<command> [options] [inputs] [outputs]

The command line may include no options and, depending on the program, 
there may be no inputs or no outputs. Most programs expect at least 
one input however. See the specific documentation 
(http://wipal.lip6.fr/documentation.html) for each program in order to 
know how many inputs and outputs each program expects.

Inputs, outputs, and options may be mixed on the command line, e.g.:

     wipal-simple-merge -n -P input1.pcap input2.pcap output.pcap
     wipal-simple-merge input1.pcap input2.pcap output.pcap -P -n
     wipal-simple-merge input1.pcap -n input2.pcap -P output.pcap
     ...

are all equivalent.

WiPal's programs use getopt(3) to parse options, so they only have short options 
(no long options) composed of a dash followed by a letter (e.g. -a, -t, etc.) 
Option letters always have the same meaning whatever the program. All options are 
not available for all programs though (some options do not make sens with some programs). 
For instance, -P always means the invoked program should consider frames with 
non-zero Prism fields as invalid. In order to know which options a program accept, 
use the -h option.

Finally, some options expect an extra argument right after they are provided:

     wipal-test-uniqueness -a hsh_80211 input.pcap
                              ^^^^^^^^^
                              This is not an input


- Basic usage

You may provide the name of a PCAP trace file as input.

          wipal-cat input.pcap output.pcap


- Input concatenation

You may provide the name of several PCAP traces separated with columns 
(do not include any space). This tells the program to consider the concatenation 
of each trace as a single input.

          wipal-cat input1.pcap:input2.pcap:input3.pcap output.pcap

will put into output.pcap the content of input1.pcap, followed by the content of 
input2.pcap and then input3.pcap.

Every program understands this syntax. Note that specifying multiple traces 
with columns makes no sense for outputs:

          wipal-cat input1.pcap:input2.pcap output1.pcap:output2.pcap

will concatenate input1.pcap and input2.pcap into a single file named output1.pcap:output2.pcap! 


- Address specification

Some programs (e.g. wipal-merge with attributes hsh_en2) might need the IPv4 
address of the machine that generated a trace to work properly. Attach such an address 
to a trace as follows:

          wipal-merge -a hsh_en2 foo.pcap=192.168.1.1 bar1.pcap:bar2.pcap=192.168.1.2

The rationale is that, in some cases, timestamps of emitted frames are not as precise 
as timestamps of received frames, and thus emitted frames should be ignored during 
synchronization. 


- Special characters

When your traces' filenames contain the special characters : or = they need to be 
escaped with a backslash (\):

          wipal-cat weird\=file\:name.pcap out.pcap
          wipal-merge -a hsh_en2 weird\=1:weird\:2=192.168.1.1 foo.pcap=192.168.1.2

1. Concatenation (and Prism noise filtering)

One may concatenate traces using the wipal-cat command. It takes exactly one input 
and one output. It may be useful to recombine a trace that was split, or filter out 
frames with Prism noise (using the -P option).

     wipal-cat in.pcap out.pcap
     wipal-cat foo.pcap.0:foo.pcap.1 foo.pcap
     wipal-cat -P in.pcap out.pcap
     wipal-cat -P bar.pcap.0:bar.pcap.1:bar.pcap.2 bar.pcap

The first example just copies in.pcap into out.pcap. Note that the two files might be 
different at the byte level, e.g. if in.pcap is big endian and the program is run 
on a little endian machine.

The second example concatenate foo.pcap.0 and foo.pcap.1 and put the result into foo.pcap.

The third example copies in.pcap into out.pcap but removes frames that have a non-zero 
noise field in their Prism headers.

The fourth example both concatenates traces while filtering noisy frames out.



2. Comparisons

One may test two PCAP traces for equivalence using the wipal-cmp command. 
The default is to compare every bit of information (PCAP headers plus packet bytes) 
but you may change this behavior using the -8, -b, -o, or -t options. Note that 
this is different however to using diff or cmp since traces with different endianness 
may contain the same packets.

By default wipal-cmp produces a report on the standard output indicating either 
that traces are equal, either which packet is the first to mismatch. 
Use -q if you are only interested in the program's exit status and do not want 
to produce any output. Use -o if you are interested in counting the number of 
bytes that differ between two traces.

e.g.:

     wipal-cmp foo.pcap bar.pcap
     wipal-cmp -q foo.pcap bar.pcap
     wipal-cmp -q -8 in1.pcap.0:in1.pcap.1 in2.pcap
     ...


3. Sub-traces

One may extract sub-traces of PCAP traces using wipal-extract-subtrace, 
wipal-extract-transmitter, or wipal-extract-bssid.

wipal-extract-subtrace takes two dates and a PCAP trace as inputs, and 
produces one output. Unfortunately, it does not support any option currently. 

wipal-extract-transmitter takes a MAC address and a PCAP trace as input, and 
produces one output. Its output contains the frames from its input that were 
transmitted by the given address. Note that the command looks at transmitters, 
not originators, e.g. the transmitter of a data frame that crossed the distribution 
system is the output access point, not the original sender. Also note that some 
frames do not contain information regarding their transmitters (e.g. MAC acknowledgements) 
and therefore cannot appear in the output, even if they were effectively sent 
by the given address. 

wipal-extract-bssid works as wipal-extract-transmitter, but the MAC address 
represents a BSSID and the command extracts frames that belong to the corresponding 
BSS. Again, note that some frames do not contain information regarding their BSS. 
These frames therefore cannot appear in the output, even if they were effectively 
belonging to the given BSS.

e.g.:
     wipal-extract-subtrace 2007-01-01 2008-01-01 \
             in.pcap.0:in.pcap.1 out.pcap
     
     wipal-extract-subtrace \
             "2004-Aug-30 16:59:39.789221" "2004-Aug-30 16:59:39.929872" \
             kalahari-ath2 subtrace.pcap
     
     wipal-extract-transmitter 71:19:9f:6f:71:33 in.pcap out.pcap
     wipal-extract-bssid       9b:d2:d7:7f:aa:63 in.pcap out.pcap



4. Merging

One may merge two IEEE 802.11 traces into one using the wipal-simple-merge command.

Use the -h option to have a description of the command's syntax. 
It takes two inputs and produce one output. When ran, the merging process starts 
by synchronizing precisely both inputs. Then both traces are merged and special care 
is given not to re-order packets or account duplicate packets twice in the output 
(that is, packets that are present in both traces appear only once in the output).

This command expects PCAP traces with either Prism headers, AVS headers, 
Radiotap headers, raw IEEE 802.11 frames, or pseudo-Ethernet II frames 
as link type. The -p and -P options only work with Prism headers. 

The following timestamps are used, unless -y is provided:

IEEE 802.11 frames
	PCAP timestamps, 
Ethernet II frames
	PCAP timestamps, 
Radiotap headers
	Radiotap headers' tsft fields. The command will fail with Radiotap headers that do not contain such fields, 
AVS headers
	AVS headers' mactime fields, 
Prism headers
	Prism headers' mactime fields.


e.g.:

     wipal-simple-merge a.pcap b.pcap output.pcap
     wipal-simple-merge -P -n foo-ath2.0:foo-ath2.1 bar-ath2 foo-bar-ath2
     ...


5. Synchronization

In order to merge two IEEE 802.11 traces WiPal needs to synchronize them precisely. 
In order to do so, it first identifies some frames that appear in both inputs. 
These are reference frames. It uses these frames to model clock desynchronization 
among the traces. It then update the first trace's timestamps so they are 
synchronized with the second trace.

One may use the wipal-synchronize command to synchronize two traces. It takes 
two inputs and produce one output. The output contains the same packets 
as the first input, but with synchronized timestamps.

To extract reference frames WiPal extract some specific frames called unique frames 
from both input traces and then intersect the two obtained sets. One may use 
the wipal-intersect-unique-frames command to get the result of this operation 
(i.e. the list of reference frames used for synchronization of two traces).

WiPal's synchronization process synchronizes reference frames before it 
synchronizes other frames. One may get the result of this operation using 
the wipal-synchronize-unique-frames command.

e.g.:

     wipal-intersect-unique-frames -n -P foo.0:foo.1:foo.2 bar.0:bar.1
     wipal-synchronize-unique-frames -n -P foo.0:foo.1:foo.2 bar.0:bar.1
     wipal-synchronize -n -P foo.0:foo.1:foo.2 bar.0:bar.1 foo-sync



6. Unique frames

A frame is said to be unique when it appears on the air once and only once 
for the whole duration of a trace. WiPal's unique frame extraction process is 
an important stage of its trace synchronization process. WiPal programs' default 
policy is to consider all beacon frames and all non-retransmitted probe 
responses as unique frames.

One may use the wipal-extract-unique-frames command to get a list of the unique 
frames that compose a trace. Run wipal-extract-unique-frames -h to get its invocation syntax.

In practice, WiPal does not extract and load full unique frames into memory. 
This would slow the process down and require an excessive amount of memory. 
The default is to work on MD5 frame hashes when WiPal was compiled using OpenSSL. 
When compiled without OpenSSL, WiPal only extracts a subset of frame fields. 
We call the pieces of information WiPal extracts to identify unique frames 
“frame attributes”, or sometimes “frame identifiers”.

You may specify frame attributes to use with the -a option. In practice, 
the difference in speed and memory consumption between attributes is negligible. 
There is an important difference between attributes, though. With some attributes, 
different unique frames may yield identical attributes (collisions). This is 
of course an undesirable behavior.

One may check that a given trace's unique frames are really unique w.r.t. 
unique frame attributes using the wipal-test-uniqueness command. 
This command finds collisions inside its input traces. 
You might specify different frame attributes using the -a option.

e.g.:

     wipal-test-uniqueness -P -a tmp foo.pcap.1:foo.pcap.2
     wipal-extract-unique-frames -P foo.pcap.1:foo.pcap.2 > foo-unique.txt


7. Duplicate data frames

One may use the wipal-find-data-dups command to search some invalid data frames. 
It looks into traces on a per-sender basis for successive duplicate data frames 
(it only considers non-retransmitted frames). Such cases should not occur in theory 
-- as it ignores retransmissions, successive data frames from the same sender should 
at least show variations in their sequence numbers. Surprisingly, some traces contain 
such anomalies: identical data frames that are not retransmissions and are only 
spaced by a few milliseconds. We have no explanations why some datasets exhibit 
those phenomena.

e.g.:

     wipal-find-data-dups foo.pcap.0:foo.pcap.1:foo.pcap.2



8. Statistics

wipal-stats computes several figures concerning its given input PCAP traces. 
It displays these figures as plain text on the standard output. You might either 
interpret them directly or post-process them with some tools, e.g. to generate plots.

(You can see the manual at http://wipal.lip6.fr/doc/wipal.html/ for more details.)


9. Anonymization

wipal-anonymize is a program to anonymize IEEE 802.11 traces. It takes one input 
and one output: the filename of the trace to anonymize, and the filename of 
the anonymous trace to produce. The output contains the same frames as the input 
with the following modifications:

- NIC specific parts of MAC addresses are anonymized.

- ESSIDs are anonymized with a prefix-preserving scheme. For instance, a valid anonymization 
could map ‘operator-4251’, ‘operator-DODO’, and ‘foobar’ to ‘abcdefgh*x0yz’, ‘abcdefgh*9876’, 
and ‘zxycba’. The anonymization scheme also preserves character classes, i.e. alpha-numerical 
characters are anonymized to other alpha-numerical characters, printable characters stay 
printable, and ASCII extended characters (128 to 256) stay extended.

- Data frames are truncated so the output only contains MAC headers.

wipal-anonymize stores valid-to-anonymous MAC and ESSID mappings into files so these mappings 
can be re-used latter. wipal-anonymize also reads these file at start-up when they exist. 
This enables the creation of distinct anonymous traces with consistent MAC addresses and ESSIDs. 
By default these mapping files' names are MAC.map and ESSID.map. Use the -m and -s options 
to change this.

[Author] Thomas Claveirole

top

[Author] Marcelo Dias de Amorim

top