Internet Monitoring With SARG and Squid On Windows

SARG aka Squid Analysis Report Generator is a nice little tool that analyses Squid's access.log files and creates a management friendly report along with graph.

This guide is for someone who have already configured SARG on Linux box.

1. Install Squid on Windows. You can follow this post squid-on-windows
2. SARG port for windows is available in the SARG website itself SARG. Or download it by clicking here.
3. Unzip it in C:\sarg
4. Edit sarg.conf file inside c:\sarg\etc
   1. Edit access_log
   2. graph option
   3. output_dir where your web server can be pointed.
   4. Others that you think is important. This is very simple file with all needed description.
5. Make your scheduled task to run SARG and rotate Squid.

Squid On Windows

Here is a short and simple guide for installing Squid on Windows for those who knows how it works in Linux.

1. Windows port for Squid is available from Acme Consulting. Grab it from http://squid.acmeconsulting.it/
2. Unzip it and put it in C drive making the folder structure c:\squid
3. You'll find squid.conf file in C:\squid\etc\squid.conf
4. Configure it as per your requirements. (mainly the following)
   1. Your hostname
   2. Dns server
   3. Cache directory
   4. ACL
   5. logrotate
      (make sure that the options you choose are available with the squid you chose by running squid -v)
5. Create the cache directory by:
1. Open Command Prompt
2. Go to C:\squid\sbin
3. Run squid.exe -D -z
6. Install the squid service by running squid.exe -i
7. You can manage squid service from services.msc or using the squid.exe CLI. Unlike Linux you may need to pass "-n Squid" as parameters for few commands.
8. Set scheduled task to rotate your log.
9. You may need to use third party software to transparently use this Squid. Until then configure the browsers to use this Squid.

Squid Access.log Meaning Explained

Squid access log is very informative if you know how to dig something out of it. Following is an a line from access.log file.

1201172176.719 1190 127.0.0.1 TCP_MISS/200 529 GET http://www.blogger.com/status.g? - DIRECT/72.14.221.191 application/xml

This line can be written as:

Timestamp, Total time, Source, Action/Code, Size, Method, URL, Ident, Hierarchy/From, Content type

1:	Timestamp	Time when the request was completed.
2:	Total time	Total time taken to complete the request
3:	Source	IP address of the client
4:	Action/Code	Action taken for the request
5:	Size	Total size of the request in bytes
6:	Method	Whether the request was GET or POST
7:	URL	The actual request
8:	Ident	Usually -
9:	Hierarchy/From	How the object is fetched and from where
10:	Content type	Type of object

Action

"TCP_" refers to requests on the HTTP port (3128)
TCP_HIT	A valid copy of the requested object was in the cache.
TCP_MISS	The requested object was not in the cache.
TCP_REFRESH_HIT	An expired copy of the requested object was in the cache. Squid made an If-Modified-Since request and the response was "Not Modified."
TCP_REFRESH_FAIL_HIT	An expired copy of the requested object was in the cache. Squid attempted to make an If-Modified-Since request, but it failed. The old (stale) object was delivered to the client.
TCP_REFRESH_MISS	An expired copy of the requested object was in the cache. Squid made an If-Modified-Since request and received a new, different object.
TCP_CLIENT_REFRESH	The client issued a request with the "no-cache" pragma. ("reload" - handled as MISS)
TCP_IMS_HIT	An If-Modified-Since GET request was received from the client. A valid copy of the object was in the cache (fresh).
TCP_IMS_MISS	An If-Modified-Since GET request was received from the client. The requested object was not in the cache (stale).
TCP_SWAPFAIL	The object was believed to be in the cache, but could not be accessed.
TCP_DENIED	Access was denied for this request.

"UDP_" refers to requests on the ICP port (3130)
UDP_HIT	A valid copy of the requested object was in the cache
UDP_HIT_OBJ	Same as UDP_HIT, but the object data was small enough to be sent in the UDP reply packet. Saves the following TCP request.
UDP_MISS	The requested object was not in the cache
UDP_DENIED	Access was denied for this request
UDP_INVALID	An invalid request was received.
UDP_RELOADING	The neighbor cache is reloading its disk store metadata and does not want any TCP requests for MISSES until it is finished.

Errors
ERR_READ_TIMEOUT	The remote site or network is unreachable - may be down.
ERR_LIFETIME_EXP	The remote site or network may be too slow or down.
ERR_NO_CLIENTS_BIG_OBJ	All Clients went away before tranmission completed and the object is too big to cache.
ERR_READ_ERROR	The remote site or network may be down.
ERR_CLIENT_ABORT	Client dropped connection before transmission completed. Squid fetches the Object according to its settings for `quick_abort'.
ERR_CONNECT_FAIL	The remote site or server may be down.
ERR_INVALID_REQ	Invalid HTTP request
ERR_UNSUP_REQ	Unsupported request
ERR_INVALID_URL	Invalid URL syntax
ERR_NO_FDS	Out of file descriptors
ERR_DNS_FAIL	DNS name lookup failure
ERR_NOT_IMPLEMENTED	Protocol Not Supported
ERR_CANNOT_FETCH	The requested URL can not currently be retrieved.
ERR_NO_RELAY	There is no WAIS relay host defined for this cache.
ERR_DISK_IO	The system disk is out of space or failing.
ERR_ZERO_SIZE_OBJECT	The remote server closed the connection before sending any data.
ERR_FTP_DISABLED	This cache is configured to NOT retrieve FTP objects.
ERR_PROXY_DENIED	Access Denied. The user must authenticate himself before accessing this cache.

CODE

Code											Reason phrase	RFC 2616 section
0											No Response Received (Squid-specific)	N/A
1xx											Informational	10.1
100											Continue	10.1.1
101											Switching Protocols	10.1.2
2xx											Successful	10.2
200											OK	10.2.1
201											Created	10.2.2
202											Accepted	10.2.3
203											Non-Authoritative Information	10.2.4
204											No Content	10.2.5
205											Reset Content	10.2.6
206											Partial Content	10.2.7
3xx											Redirection	10.3
300											Multiple Choices	10.3.1
301											Moved Permanently	10.3.2
302											Found	10.3.3
303											See Other	10.3.4
304											Not Modified	10.3.5
305											Use Proxy	10.3.6
306											(Unused)	10.3.7
307											Temporary Redirect	10.3.8
4xx											Client Error	10.4
400											Bad Request	10.4.1
401											Unauthorized	10.4.2
402											Payment Required	10.4.3
403											Forbidden	10.4.4
404											Not Found	10.4.5
405											Method Not Allowed	10.4.6
406											Not Acceptable	10.4.7
407											Proxy Authentication Required	10.4.8
408											Request Timeout	10.4.9
409											Conflict	10.4.10
410											Gone	10.4.11
411											Length Required	10.4.12
412											Precondition Failed	10.4.13
413											Request Entity Too Large	10.4.14
414											Request-URI Too Long	10.4.15
415											Unsupported Media Type	10.4.16
416											Requested Range Not Satisfiable	10.4.17
417											Expectation Failed	10.4.18
5xx											Server Error	10.5
500											Internal Server Error	10.5.1
501											Not Implemented	10.5.2
502											Bad Gateway	10.5.3
503											Service Unavailable	10.5.4
504											Gateway Timeout	10.5.5
505											HTTP Version Not Supported	10.5.6
6xx											Proxy Error	N/A
600											Unparseable Response Headers (Squid-specific)	N/A

Methods

GET	Request URL
HEAD	Request only HTTP headers of the supplied URL and no document body
POST	Transfer data to the supplied URL
PUT	Store data under the supplied URL
CONNECT	Forward data to SSL-Server:Port
ICP_QUERY	Request from a Parent/Neighbor for the supplied URL
NONE	Request of an unsupported method

Hierarchy

NONE	The object requested by a sibling, was not in my cache.
DIRECT	The object has been requested from the origin server.
SIBLING_HIT	The object was requested from a neighbor cache which replied with a UDP_HIT (formerly logged as NEIGHBOR_HIT).
PARENT_HIT	The object was requested from a parent cache which replied with a UDP_HIT.
DEFAULT_PARENT	The object was requested from a default parent cache appropriate for this URL.
SINGLE_PARENT	The object was requested from the only parent cache appropriate for this URL.
FIRST_UP_PARENT	The object has been requested from the first available parent in your list.
NO_PARENT_DIRECT	The object was requested from the origin server because no parent caches exist for the URL.
FIRST_PARENT_MISS	The object has been requested from the parent cache with the fastest weighted round trip time.
ROUNDROBIN_PARENT	No ICP queries were received from any parent caches. This parent was chosen because it was marked as 'default' in the config file and it had the lowest round-robin use count.
CLOSEST_PARENT_MISS	This parent was selected because it included the lowest RTT measurement to the origin server. This only appears with 'query_icmp on' set in the config file.
CLOSEST_DIRECT	The object was fetched directly from the origin server because this cache measured a lower RTT than any of the parent caches.
LOCAL_IP_DIRECT	The object has been requested from the origin server because the origin host IP address matched your 'local_ip' list.
FIREWALL_IP_DIRECT	The object has been requested from the origin server because the origin host IP address is inside your firewall.
NO_DIRECT_FAIL	The object could not be requested because of firewall restrictions and no parent caches were available.
SOURCE_FASTEST	The object was requested from the origin server because the 'source_ping' reply arrived first.
SIBLING_UDP_HIT_OBJ	The object was received in a UDP_HIT_OBJ reply from a neighbor cache (formerly logged as UDP_HIT_OBJ).
PARENT_UDP_HIT_OBJ	The object was received in a UDP_HIT_OBJ reply from a parent cache (formerly logged as UDP_HIT_OBJ).
PASSTHROUGH_PARENT	The neighbor or proxy defined in the config option 'passthrough_proxy' was used.
SSL_PARENT_MISS	The neighbor or proxy defined in the config option 'ssl_proxy' was used.

Remove Squid footer completely

You can remove footer from squid error pages. Removing the version number from the footer is very easy.

In squid.conf enable
httpd_suppress_version_string

But what if you want to remove the whole message.
Generated Thu, 08 Jun 2000 06:58:30 GMT by proxy1.proxy.com (Squid/2.3.STABLE1)
There is no configuration available to remove it but you can always edit the source file for that. Don't worry its very easy. Easy because its really easy.

Just follow the following steps:

1. Go to squid source. /usr/src/squid/src
2. Backup the existing file errorpage.c cp ./errorpage.c ./errorpage.c.bkp
3. Remove errorpage.o rm errorpage.o
   
4. Edit the file errorpage.c vim errorpage.c
   

   {
   ERR_SQUID_SIGNATURE,
   "\n<\BR clear=\"all\">\n"
   "<\HR noshade size=\"1px\">\n"
   "

   \n"
   "Generated %T by %h (%s)\n"
   "

   \n"
   "<\/BODY><\/HTML>\n"
   }
   

5. Remove the line "Generated %T by %h (%s)\n"
6. Save and exit :wq
7. Reconfigure squid with old options ./configure .........
   
8. Make clean make clean
9. Make make
10. Make all make all
11. Make install make install
12. Stop squid /usr/local/squid/sbin/squid -k shutdown
13. Start squid /usr/local/squid/sbin

SARG - Squid Analysis Report Generator

SARG - Squid Analysis Report Generator is a tool that allow you to view "where" your users are going to on the Internet.

Its very useful tool which generates graphical and tabular report from your access.log file in a web browser.







You can get sarg from http://sarg.sourceforge.net/sarg.php
Extract it to your src folder, configure it and install it.

The configuration file is sarg.conf. You really don't need to change that file as most of the option can be passed through command line parameters. One thing you must make sure is that the output directory is your web accessible directory. The configuration file is pretty simple. Also make sure that it gets your right access.log file.

MRTG for SQUID

MRTG for SQUID gives you a detail idea of what is happening behind the scene. In this tutorial I'll make sure that you will successfully set up one like below.


First thing is to make sure that you have given --enable-snmp option while compiling your squid. If you have not done so then its time to do so. Compile your squid with --enable-snmp and then 'make clean', 'make', and 'make install'

Edit you squid.conf file so that
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic localhost
snmp_access deny all

The first line defines the port SNMP is listening on squid.
Second line defines public as the community string or lets say password.
Third line defines that from localhost snmp query can be made to squid.
Fourth line denies query from all.(except localhost)

Now you need a web server to serve the graphs.

Download apache from http://www.reverse.net/pub/apache/httpd/httpd-2.2.6.tar.bz2
mv ./httpd-2.2.6.tar.bz2 /usr/local/src
cd /usr/local/src
bunzip2 -c /usr/local/src/httpd-2.2.6.tar.bz2 | tar xvf -
rm httpd-2.2.6.tar.bz2
mv httpd-* httpd
cd httpd
./configure --enable-prefix=/usr/local/httpd
make
make install

Now you need to configure apache. Lets just host it in a simple way.

mkdir -p /usr/local/www/mrtg
vi /us/local/httpd/conf/httpd.conf
Change DocumentRoot to "/usr/local/www/mrtg"
save and exit
run /usr/local/httpd/bin/apachectl -k start

Now its time for setting MRTG server. Download the following programs:

http://www.gzip.org/zlib (gzip
http://www.libpng.org/pub/png/libpng.html (libpng)
http://www.libgd.org/ (gd)
http://oss.oetiker.ch/mrtg/pub (mrtg)

Copy zlib to /usr/local/src
gunzip -c zlib-*.tar.gz | tar xvf -
rm zlib-*.tar.gz
mv zlib-* zlib
cd zlib
./configure
make
cd ..

Copy libppng to /usr/local/src
gunzip -c libpng-*.tar.gz |tar xvf -
rm libpng-*.tar.gz
mv libpng-* libpng
cd libpng
make -f scripts/makefile.std CC=gcc ZLIBLIB=../zlib ZLIBINC=../zlib
rm *.so.* *.so
cd ..

The \ in the following line means that they are same line.

Copy gd to /usr/local/src
gunzip -c gd-2.0.11.tar.gz |tar xf -
mv gd-2.0.11 gd
cd gd
env CPPFLAGS="-I../zlib -I../libpng" LDFLAGS="-L../zlib -L../libpng" \
./configure --disable-shared --without-freetype --without-jpeg
make
cp .libs/* .
cd ..

Copy mrtg to /usr/local/src
gunzip -c mrtg-2.15.2.tar.gz | tar xvf -
cd mrtg-2.15.2
./configure --prefix=/usr/local/mrtg-2 \
--with-gd=/usr/local/src/gd \
--with-z=/usr/local/src/zlib \
--with-png=/usr/local/src/libpng
make
make install
mkdir -p /usr/localmrtg-2/cfg

Now lets make graph for squid.

Download the following two files

mrtg-squid.cfg
squid.mib
Copy them to /usr/local/mrtg-2/cfg/
edit mrtg-squid.cfg and do as the file says. Put working dir as /var/www/mrtg

Lets make graph
/usr/local/mrtg-2/bin/mrtg /usr/local/mrtg-2/cfg/mrtg-squid.cfg
Run this for three times. Its OK for now to see few errors.

Lets make index page
/usr/locl/mrtg-2/bin/indexmaker --title="OSR" --show=day --section=title --output=/var/www/mrtg/index.html /usr/local/mrtg-2/cfg/mrtg-squid.cfg

Fire your browser and you should see the graph

For continous monitoring
crontab -e
*/5 * * * * /usr/local/mrtg-2/bin/mrtg /usr/local/mrtg-2/cfg/mrtg-squid.cfg --logging=/var/log/mrtg.log

Additional reading: http://wiki.squid-cache.org/SquidFaq/SquidSnmp