FutureQuest, Inc. FutureQuest, Inc. FutureQuest, Inc.
Knowledgebase: Stats/Logs
Raw referrer logs versus access logs?
Posted on 31 January 2004 05:39 AM
Question:

I asked where I could find the raw referrer logs for my domain, and was told that I could find the raw logs at:
/big/dom/xMyDomain/logs_web

However, I see only access logs. The entries in the access logs do not show the referring page (or even the domain), which I am seeking.

Viewing the "All Refers" section of my stats, it shows the referring URLs. I presume this page is compiled from raw referrer logs, but the access logs (eg access.today) in /big/dom/xdomain/logs_web do not include domain names. This leads me to believe that the access logs are not the referrer logs.

Answer:

For efficiency sake: minimize filehandles and performance reasons: syscall reduction

We do not maintain separate 'access' and 'referrer' logs...

We use the industry standardized: NCSA's extended/combined log format.

ELF: "%h %l %u %t \\"%r\\" %>s %b \\"%{Referer}i\\" \\"%{User-agent}i\\""

ELF == Extended Log Format

As you can see from the line above, your 'Referer' is included...

Technically, it's a pain to efficiently parse out for various reasons - but it is what most all log file analyzers are compatible with...

We would highly recommend reading the following page for an interpretation of the other fields:
http://httpd.apache.org/docs/mod/mod_log_config.html

As a premonition, the above ELF format cannot be changed/adjusted/altered for any reason.

The "All Refers" page is the processing results from the stats run every night which builds those files from the raw logs. If you want something similar, then you will need to write your own custom raw log file processing program to extract the results you need...

Hope this helps to clear up any confusion.