Quote:
Originally Posted by kae 3000 tables, eh?
there's such a thing as over-normalisation, you know! |
Yep but there is also such a thing as computability.

This is one of the smaller dbs for tracking new/deleted/moved domains in com/net/org/biz/info/ie/eu/uk zones from 2000 to 2007. I might have to take a look at optimising the design somewhat but it was designed to handle large amounts of data with the emphasis on computability - dumping a single .com zone's domains and nameservers into a table would result in a table of about 133 Million rows. So even comparing two 133M row tables would take some time. Given the way that the .com has mushroomed over the past year (from about 3G to 4.3G of raw data), the design has held up fairly well. Though it only takes a few seconds to process .ie stats.
I ran a summary table generation today to coallate the stats on all hosters (approximately 1.5 Million) in com/net/org/biz/info/ie since December 2000 to December 2006 and it took approximately 9 hours on an Athlon64 with 3G of RAM. Even with optimisation, big sets of data take a lot of time to process.
Regards...jmcc