We are interested in keeping logs in RDBMS.What are the indexing
schemes that we can apply for huge raw log data? The purpose is to use
this log information for datamining. How can we mine patterns and
correlations in database efficiently ? Please give any research papers
or links to refer.

---

Clarification of Question by mr_niceguy-ga on 08 Nov 2005 17:08 PST

By log I mean bunch of transaction data. The logs could be
security-related syslogs or email traffic or records or people seeing
online ads etc

what do you mean "logs", the term log has special meaning in database
world, such as event log or transaction log... i assume that you mean
a bunch of transaction data?

indexing scheme is determined by what kind of data scheme you have and
what kind of query you will submit. there are some general rules,
however, most of the time, they need to be fine tuned. same thing for
the data mining stuff.

You should find something here.
http://www.loganalysis.org/

I do regular analysis on about 1000 syslogs (all collected in one
file) using Perl.

Oh - and syslog s horrible as every different version of any program
makes it's messages in a differnet format.  I'd almost prefer an
SNMP-style encoding.

You know your analysis is doing some good when people call you to
explain the funny things they've been doing rather than having you see
it first in the logs. :)