User's Guide to the Webfwlog Report Editor

Navigation Buttons

Refresh

Press the Refresh button to update the display. This is useful, for example, if you have added
or removed columns, or changed their order, and want to see the list updated reflecting the changes.
You should also do this before running a report if you want to be able to reuse the settings in the
same session by using the back button to return to the report editor, but do not want to save the
new settings permanently.

Run Report

Press the Run Report button to run a report using the current settings. You should press the Refresh
button first if you want any changes to the settings remembered if you use your browser's back button
button to return to the report editor.

Return to Main Menu

Press the Return to Main Menu button to go to the main menu.

Report Management

Select Report

You can recall the settings for a previously saved report by selecting it from the list and
pressing the Use Report button. Any unsaved changes to the current report will be lost.

Status

Status Messages are displayed below the Select Report Control.

Report Code

This is the code of the saved report currently being edited.
If the report has not yet been saved it will show as "<NEW>"
If an imported report has not been saved it will show as "<name_of_imported_file>"

Description

This is a description of the current report and will be displayed on the main menu.
It will be saved with the report definition.

Save

Press the Save button to save the current report definition. You will be prompted for a report
code or to confirm overwriting an existing definition.

Delete

Press Delete to permanently delete the report definition. You will be prompted to confirm this action.

Save as Default

Press Save as Default to save the current settings as the default to be used when creating
new reports. There are some built-in defaults that will be used if you have not saved
any default settings.

Create New

Press the Create New button to create a new report using the default settings. Any unsaved
changes to the current report will be lost.

Export

Press the Export button to export the current report settings to a file. You will be
prompted for a filename.

Import

Press the Import button to import settings from a file. You will be prompted for a
filename, or can browse for the file you want to use.

Report Options

Data Source

Select the data source for this report. Default will use the default data source specified in
webfwlog.conf. Database will use data logged in a database using the ULOG or NFLOG target of linux
netfilter, Snort IDS or other database logs. You can also select one or more tables or views to use,
overriding the default specified in webfwlog.conf. Syslog will use system log files.

File Name

For syslog data, enter the filename(s) to parse separated by spaces.
Multiple files can also be specified using the syntax of the shell available to PHP,
e.g., messages{,.1.gz}

Title

Enter a title for this report which will appear in the report heading.

Rows per page

This is the number of rows that will appear on each page of output. Enter 0 or leave blank
to display all rows.

Page Refresh Rate

The report will refresh at this interval in seconds if your browser supports meta-refresh.
Setting to zero disables refresh.

Update Hostname Cache

Check the Update Hostname Cache box to update the hostname and services caches for hostnames
and services appearing in report every time the report is run, but be aware that this may dramatically
increase the time needed to run a report. This setting is saved with the report definition.

Populate Hostname Cache

Check the Populate Cache Full box to populate the hostname and services caches for hostnames
and services appearing in report every time the report is run, but without resolving the new entries.
The caches can then be updated using the Update Cache Button from the main page or from the
report editor at a later time. This setting is saved with the report definition.

Populate Cache Full

Check the Populate Hostname Cache box to populate the hostname and services caches for hostnames
and services in report every time the report is run, including hostnames and services for selected rows
not appearing in a limited result, and without resolving the new entries. The caches can then be updated
using the Update Cache Button from the main page or from the report editor at a later time. This setting is
saved with the report definition.

Update Cache button

Press the Update Cache button to update the hostname and services caches immediately.

Please be aware that this may take a LONG time depending upon how many new unique IPs are
found. You may cancel this operation at any time, and any hostnames already
updated will be saved and not need to be updated again.

Columns to Include Menu

Columns to Include

You may include any columns you wish in the report and in any order. To include a column in the
report enter a number in the box next to the column description. The columns will appear in the
report in numerical order as entered. To remove a column from the report make the box blank. If you
wish to insert a column between two columns you can enter a decimal fraction, e.g. 3.5 will go between
3 and 4. You can enter 0 to place a column first. If two columns have the same number the order in
which they will appear is not defined.

Only some of all possible columns are shown on the report editing menu. To see the remaining columns
press the More button. Pressing the Refresh button will update the display of the columns in current
numerical order, and renumber the columns beginning with 1.

Not all of the columns shown will be available for every log format. If you select a column that
is not in your database logs an error will occur; if your logs are in files the column will be blank or 0
depending upon the type of column, or left out entirely.

Links

Check the link box next to a column to include a hyperlink in each cell for that column in the
report. Generally, clicking on a link in a report will filter the report by the item in the cell
selected. For example, if you have a report that has tcp, udp and icmp packets included, clicking on
a link in the protocol column that shows "tcp" will redisplay the report with all other settings the
same but showing only rows for tcp packets. You can further filter the report by clicking on a link
in another column. Continuing the example, clicking on a link in the source IP column will filter the
report by the source IP selected, and you will now see a report showing only tcp packets from that
source IP.

One exception to this is the Count column on summarized reports. Clicking on a link in this column
will show an unsummarized report showing the individual logged entries that make up that line. For
example, if the count column shows "12" clicking on the link will show the 12 logged entries.

The other exception is the Packet column, which shows a unique auto-generated id for each logged
packet in database log, or the line number for file logs. Clicking on a link in the Packet column will
show all the details for that packet whether or not they appear in the report.

Certain columns do not have checkboxes for links because links would not be meaningful. These include
all date/time columns as well as looked-up columns such as hostnames and service names.

There are also always links in the column headings on reports. Clicking on a link in a column heading
will sort the report by that column. Clicking on the column heading again will reverse the order in
which the column is sorted.

Arbitrary Column

If your data is logged to a database, you can enter an arbitrary column definition in the blank text box and
this will be added to SQL SELECT statement as-is. Checking the "Grp" box will add this column to
the GROUP BY clause, which MUST be selected if the column definition does not contain aggregate functions
such as max(), min() or sum(), and MUST NOT be selected otherwise. The summarize box should also be selected
if this definition includes aggregate functions.

It is up to you to ensure this is a valid SQL column definition for your database server, that Grp is
selected if needed, and that a link for drill-down, if selected, makes sense.

If you data is logged to files this control has no effect.

Sort Order Menu

Summarize Report

Check the Summarize Report box to group the report by the columns selected for the report. For example,
if you have a report with the Protocol, Source IP and Destination port columns included, and have 15
logged packets from one source IP to tcp port 25, and 34 from another source IP to tcp port 25, you
would get a two-line report, with 15 and 34 appearing the Count column, which you would normally
include in a summarized report

For database logs, summarizing a report means invoking the "GROUP BY" clause of a SQL query, and you
cannot include an aggregated column in an unsummarized report, such as Count, Earliest and Latest. You will
receive an error message if you try to do so. All other selected columns will be included in the GROUP BY
clause.

Sort Order

The report will be sorted in the order shown here. To change to sort order of a report, put a number
in the box next to the column. The columns will be sorted in the report in numerical order as entered.
To remove a column from the sort order make the box blank. If you wish to insert a column into the sort
order between two columns you can enter a decimal fraction, e.g. 3.5 will go between 3 and 4. You can
enter 0 to put a column first in the sort order. If two columns have the same number the order in which
they will sorted is not defined.

Pressing the Refresh button will update the display of the sort order in current numerical order,
and renumber the sort order beginning with 1.

To sort a column in reverse order check the Desc box.

Only some of all possible columns that can be included in the sort order are shown on the report editing
menu. To see the remaining columns press the More button.

Sorting by geoip fields requires necessary support. See README.geoip for more information.

Criteria Menu

Criteria Entry

Enter the values that you want to use to restrict the logged packets that will be included in the report.
Check the Inv box to invert the test and include only logged packets that do NOT match the value entered.
If your logged data do not include a field, adding selection criteria for that field may result in no
records being selected.

Dates

Dates can be entered using the PHP date and time syntax. This means that you can enter things like "yesterday"
or "last week" and it will be saved this way, meaning you can have a report that always shows you
recent activity, for example.

The Min date and Max date values are used to restrict the report to packets within the specified range.

This paragraph relates to ulog database logs only. Ulog records the timestamp of a packet in the
oob_time_sec field in the database, and as ancient linux kernels did not record a timestamp for locally
generated packets this field would be NULL or 0 for such packets. As a workaround, ulog provided the LOCAL
plugin which would record the time of the logging host in the local_time field of the database. Values
entered for Min date and Max date are matched against the local_time field if it exists, otherwise
against the oob_time_sec field. If both fields exist you can force a match against oob_time_sec by checking
the oob time box. With recent kernels the local_time field is not useful or needed, and in the absence of
this field oob_time_sec will always be used. In other words, unless you are running an ancient linux kernel
you can ignore this paragraph.

Local Host
Log Label
Input Interface
Output Interface

You can select packets based on the input and/or output interfaces, log label, and also the
local hostname using POSIX extended regular expressions for file logs, and for database logs the matching used
by the regualar expression operator for your database server. This is usually POSIX extended by default but
can be changed in some cases; see the documentation for your database server.

For example, if you want to include all packets that have DROP in the log label simply enter "DROP" in the box.
However, if you want to match only packets that have the exact text "DROP" as the log label then enter "^DROP$"
in the box.

For Netfilter, the log label is an optional user-defined argument (eg: "log-prefix") to the LOG, ULOG or NFLOG target.
For Ipfilter, the log label is "<group>:<rule number> <action> <log-tag>", where log-tag is optional. man ipmon for more info.
For Ipfw, the log label is "<rule number> <action>". Man ipfw for more information.
For Ipchains, the log prefix is the target name, such as ACCEPT or DENY.
For Snort, the log label is the classification. See the snort documentation for details.
For Cisco IOS routers, the log label is "<rule number> <action>", where action is Deny or Permit.
For Cisco PIX routers, the log label is always "Deny"
For Netscreen, the log label is the action. See Netscreen documentation.

If your packets are logged in a database, you should be aware that depending upon the column type for these
fields in your database, white space may or may not be trimmed from the beginning and end of the value stored
in the database and you need to take this into account when you formulate your regular expression. Also,
whitespace will ALWAYS be trimmed from the beginning and end of what you enter here, so you will need to use
bracket expressions if you want to begin or end your RE with white space, e.g. "^[[:space:]]+" to match
whitespace at the beginning of the value using POSIX extended RE's.

TCP Source Port
TCP Destination Port
UDP Source Port
UDP Destination Port
ICMPv4 Type
ICMPv4 Code
ICMPv6 Type
ICMPv6 Code

Enter the tcp port, udp port or icmp type and/or code you want to include in your report. You can
enter a single value, a range of values separated with a ":", or a comma-delimited list of values or
or ranges. For example, entering "22:25, 80, 110" will select packets for that field with values
from 22 to 25, or a value of 80, or a value of 110.
Note that entering a value here will only affect packets for the appropriate protocol, and all packets
for other protocols will still be included in the report. If you only want to select one protocol use
the IP Protocol selector.

IP Protocol

Enter the IP protocols you want to include as a name or number, i.e., "6" and "tcp" are equivalent.
You can enter a single protocol, a comma-delimited list or a numeric range of protocols separated by a":"

TCP Options

Check the boxes for the TCP options by which you want to select packets. Check the exact match box
if you want to select those packets which have exactly the selected options. For example, without the
exact match box, selecting SYN will include packets with just the SYN bit set, as well as those with
both the SYN and ACK bits set, etc. On the other hand, checking the exact match box would include
only those with just the SYN bit set.

Note that making selections here will only affect tcp packets. All packets for other protocols will
still be included in the report. If you only want to select tcp packets use the IP Protocol selector.

Source IP
Destination IP
ICMP Redirect Gateway

Enter the IP address you want to include, with an optional prefix, in either dotted-quad IPv4 address or any valid IPv6
address notation. You may include a prefix such as /24 or /64 to specify a address range, which for IPv4 may be
specified as a netmask, i.e., xxx.xxx.yyy.yyy/255.255.255.0 and xxx.xxx.yyy.yyy/24 are equivalent, except that if your
data is logged to a postgresql database with a column type of inet for these columns, you must use prefix notation to
specify a netmask or an error will result. You can specify a single address or a comma-delimited list of addresses.

Min Count
Max Count

For summarized reports, enter the minimum or maximum value for the Count column to include in the report

Selectors for the following criteria can be accessed by pressing the More button.

Latest Earliest
Earliest Latest

Enter the most recent value for the Earliest column or the oldest value for the Latest column that you want
to include in the report. See also discussion about dates

IP Family
IP TOS / IPv6 TC
IP TTL / IPv6 Hoplimit
IP Header Length
IP Total Length
IP ID
IP Checksum
IPv6 Flowlabel
IPv6 Payload Length
Firewall Mark
TCP Sequence Number
TCP ACK Sequence Number
TCP Window
TCP Data Offset
TCP Urgent Pointer
TCP Checksum
UDP Length
UDP Checksum
ICMPv4 Echo ID
ICMPv4 Echo Sequence Number
ICMPv4 Next hop MTU
ICMPv6 Echo ID
ICMPv6 Echo Sequence Number
(Source|Destination) Latitude
(Source|Destination) Longitude
(Source|Destination) Metro Code
(Source|Destination) Location Accuracy
Ethernet Protocol
AH/ESP protocol SPI
Time µsec

Enter the value for these fields for the packets you want to include in your report. If the entry begins
with "0x" it will be interpreted as a hexadecimal value. If the entry begin with "0b" it will be
interpreted as a binary value. If the first digit is zero, the entry will be interpreted as an octal value.
Otherwise the entry will be interpreted as a decimal value, or floating point if a '.' is present.

You can enter a single value, a range of values separated with a ":", or a comma-delimited list of values or
or ranges. For example, entering "20:100, 1000" will select packets for that field with values
from 20 to 100, or a value of 1000.

Note that entering values for selectors for particular protocols such as tcp,udp,icmp or ah/esp will only
affect packets for the appropriate protocol, and all packets for other protocols will still be included in
the report. If you only want to select one protocol use the IP Protocol selector.

IP Family should be a numeric value for AF_INET or AF_INET6 for IPv4 and IPv6, respectively. AF_INET is 2 on
most system, but AF_INET6 varies. For linux it is 10, for BSD-ish systems it is 28, 26 for Solaris, and could
be other values on different systems. Try including the IP Family column and see what values you get for IPv6
packets.

Note also that the defintions for some fields have changed over time (notably TOS), and interpretation of
such fields is implementation dependent.

Local Host
Source MAC
Destination MAC
MAC String
(Source|Destination) Continent Code
(Source|Destination) Continent
(Source|Destination) Country ISO
(Source|Destination) Country
(Source|Destination) Subdivision 1 Code
(Source|Destination) Subdivision 1 Name
(Source|Destination) Subdivision 2 Code
(Source|Destination) Subdivision 2 Name
(Source|Destination) City Name
(Source|Destination) Time Zone
(Source|Destination) Postal Code
(Source|Destination) Registered Country
(Source|Destination) Represented Country

You can select packets based on MAC address and geoip fields using POSIX extended regular expressions for
file logs, and for database logs the matching used by the regualar expression operator for your database server.
This is usually POSIX extended by default but can be changed in some cases; see the documentation for your
database server. Filtering by Geoip fields requires necessary support, see README.geoip for more information.

For example, if you want to include all packets that have DROP in the log label simply enter "DROP" in the box.
However, if you want to match only packets that have the exact text "DROP" as the log label then enter "^DROP$"
in the box.

If your packets are logged in a database, you should be aware that depending upon the column type for these
fields in your database, white space may or may not be trimmed from the beginning and end of the value stored
in the database and you need to take this into account when you formulate your regular expression. Also,
whitespace will ALWAYS be trimmed from the beginning and end of what you enter here, so you will need to use
bracket expressions if you want to begin or end your RE with white space, e.g. "^[[:space:]]+" to match
whitespace at the beginning of the value using POSIX extended RE's.

(Source|Destination) Is Anonymous Proxy?
(Source|Destination) Is Satellite Provider?
(Source|Destination) Is In European Union?

For these boolean fields, you can enter Yes or 1 or true to select where the field is true, or No or 0 or false
to select where the field is false. You can also enter -1 to select where the field is null, such as IPs for which
there is no geoip record, e.g., private addresses such as 192.168.0.0/24.

Fragmentation Offset

Check the DF or MF checkbox to include packets with the respective flag set. Enter the fragementation offset
value for packets to include in the text box. DF and MF only have meaning for IPv4 packets and are ignored for
IPv6 packets

Additional SQL WHERE clause

If you data is logged to a database, whatever you enter here will be added to the WHERE clause of the SQL
query as-is. This will be enclosed in "(" and ")" as a sub-expression and ANDed with
any other specified criteria. When specifying critera, you will need to refer to the database fieldnames
and not to the displayed column labels.

If your data is logged to files this control has no effect.

SQL HAVING clause

If you data is logged to a database, what you enter here will be added to the HAVING clause of the SQL query as-is.
You would normally put here selection criteria based on aggregate functions, such as max(), min() or sum.
For example "sum(ip_datalen)>1000." You should not put here criteria not based on aggregate functions;
these should go in the WHERE clause instead.

If your data is logged to files this control has no effect.

Miscellaneous

Hostname Cache

If you have created the hostnames table during setup, you will be able to include a column in your
reports with reverse DNS lookups for source and destination IP addresses. If the table does not exist
the hostname columns will be ignored; for performance reason Webfwlog will not attempt to resolve
hostnames without a cache.

The cache can be updated in one of two ways, every time a report is run or as a separate operation. Note
that this can take a long time if it has been awhile since you last updated the cache or have never
updated the cache

Service Name Cache

If you have created the services table during setup, you will be able to include a column in your
reports with the name of tcp and udp services, derived from the services file. If the table
does not exist the service name columns will be ignored; for performance reason Webfwlog will not
attempt to resolve service names without a cache.

The cache can be updated in one of two ways, every time a report is run or as a separate operation. Note
that this can take a long time if it has been awhile since you last updated the cache or have never
updated the cache

Field Name cross-reference

When specifying criteria in the additional WHERE clause, you must refer to field names as they exist in
database. Following is a cross-reference from the column label to the field name for ulogd:

Source Name	sname
Record Num	id
Time	oob_time_sec
Time µsec	oob_time_usec
Input Interface	oob_in
Output Interface	oob_out
Log Label	oob_prefix
Firewall Mark	oob_mark
IP Family	oob_family
IP Protocol	ip_protocol
Source IP	ip_saddr
Destination IP	ip_daddr
IP TOS / IPv6 TC	ip_tos
IP TTL / IPv6 Hoplimit	ip_ttl
IP Total Length	ip_totlen
IP Header Length	ip_ihl
IP Checksum	ip_csum
IP ID	ip_id
Fragmentation Offset	ip_fragoff
IPv6 Traffic Class	ip6_priority
IPv6 Hoplimit	ip6_hoplimit
IPv6 Flowlabel	ip6_flowlabel
IPv6 Payload Length	ip6_payloadlen
IPv6 Fragmentation ID	ip6_fragid
IPv6 Fragmentation Offset	ip6_fragoff
TCP Source Port	tcp_sport
TCP Destination Port	tcp_dport
TCP Sequence Number	tcp_seq
TCP ACK Sequence Number	tcp_ackseq
TCP Window	tcp_window
TCP Data Offset	tcp_off
TCP Checksum	tcp_csum
UDP Source Port	udp_sport
UDP Destination Port	udp_dport
UDP Length	udp_len
UDP Checksum	udp_csum
ICMP Checksum	icmpcsum
ICMPv4 Type	icmp_type
ICMPv4 Code	icmp_code
ICMPv4 Echo ID	icmp_echoid
ICMPv4 Echo Sequence Number	icmp_echoseq
ICMPv4 Redirect Gateway	icmp_gateway
ICMPv4 Next Hop MTU	icmp_fragmtu
ICMPv4 Checksum	icmp_csum
ICMPv6 Type	icmpv6_type
ICMPv6 Code	icmpv6_code
ICMPv6 Echo ID	icmpv6_echoid
ICMPv6 Echo Sequence Number	icmpv6_echoseq
ICMPv6 Checksum	icmpv6_csum
ARP HW Type	raw_type
MAC String	raw_mac
Ethernet Protocol	oob_protocol
Source MAC	mac_saddr_str
Destination MAC	mac_daddr_str
Netfilter Hook	oob_hook
Local Host	local_hostname
Source Continent Code	continent_code_s
Source Continent	continent_name_s
Source Country ISO	country_iso_code_s
Source Country	country_name_s
Source Subdivision 1 Code	subdivision_1_iso_code_s
Source Subdivision 1 Name	subdivision_1_name_s
Source Subdivision 2 Code	subdivision_2_iso_code_s
Source subdivision 2 Name	subdivision_2_name_s
Source City Name	city_name_s
Source Metro Code	metro_code_s
Source Time Zone	time_zone_s
Source Location Accuracy	accuracy_radius_s
Source Registered Country	registered_country_geoname_id_s
Source Represented Country	represented_country_geoname_id_s
Source Is Anonymous Proxy?	is_anonymous_proxy_s
Source Is Satellite Provider?	is_satellite_provider_s
Source Postal Code	postal_code_s
Source Latitude	latitude_s
Source Longitude	longitude_s
Source In European Union?	is_in_european_union_s
Source Autonomous System Number	autonomous_system_number_s
Source Autonomous System Organization	autonomous_system_organization_s
Destination Continent Code	continent_code_d
Destination Continent	continent_name_d
Destination Country ISO	country_iso_code_d
Destination Country	country_name_d
Destination Subdivision 1 Code	subdivision_1_iso_code_d
Destination Subdivision 1 Name	subdivision_1_name_d
Destination Subdivision 2 Code	subdivision_2_iso_code_d
Destination Subdivision 2 Name	subdivision_2_name_d
Destination City Name	city_name_d
Destination Metro Code	metro_code_d
Destination Time Zone	time_zone_d
Destination Location Accuracy	accuracy_radius_d
Destination Registered Country	registered_country_geoname_id_d
Destination Represented Country	represented_country_geoname_id_d
Destination Is Anonymous Proxy?	is_anonymous_proxy_d
Destination Is Satellite Provider?	is_satellite_provider_d
Destination Postal Code	postal_code_d
Destination Latitude	latitude_d
Destination Longitude	longitude_d
Destination In European Union?	is_in_european_union_d
Destination Autonomous System Number	autonomous_system_number_d
Destination Autonomous System Organization	autonomous_system_organization_d
pwsniff user	pwsniff_user
pwsniff pass	pwsniff_pass
AH/ESP protocol SPI	ahesp_spi