S3 Stats - Log files not being processed
So we have a ttr2 bucket that has logging enabled, yet the last the time log file was processed was March 8th. We have 2 other buckets that have processed the logs today.We would like to have the log files processed ASAP if that is possible.Also, I am not sure how the log files are saved, but the last log file for this bucket was 500MB. Is it possible to create log files for each bucket per day. Thanks(This is pretty high priority so if this problem could be attended to ASAP, that would be good. Thanks)
I see the same thing that you do. It actually looks like the last two days did run, but there are definitely holes like you say.I notice that the traffic to this bucket is remarkably high, with nearly 30M hits one day this month. I've seen this same sort of failure with a couple other high-traffic buckets, and I had in the past written it off to Webalizer not being able to handle that sort of load.In this last week though, I've isolated a rare issue that causes the compilation of log files to fail due to malformed log lines. It would seem to follow that high-traffic buckets would be more likely to encounter this error, and would see higher failure rates. Indeed, looking at the bucket in question it appears that there are several compiled logfiles missing, which would indicate that the fault might lie there.I'm planning to push the fix for this live over the weekend, so it's possible that this issue will resolve itself over the next week. I'll follow your bucket more closely and keep you posted.Sorry for the inconvenience.
jason
Friday, March 13, 2009
Is there a quick and dirty way to access the logs myself without changing the permissions on our buckets that S3Stat is already tracking.We just want to download the logs, but don't want that to interfere with any thing else.
A Lacy
Friday, March 13, 2009
Server Access Logging is built in to S3, so can certainly access the raw logs without giving your credentials to us. It's a bit fiddly to set up though, and the logs that it gives aren't ready to feed into most analytics packages.So I guess the answer is "Yes and No". Yes, it's certainly doable. No, it's not really quick and dirty. Here are some links to walkthroughs in various languages that might get you headed in the right direction:http://www.s3stat.com/web-stats/S3-resources.ashxGood luck!
jason
Sunday, March 15, 2009
Can you think of any reason why the numbers being reported in the s3 processed log files are different from the numbers I am getting if i download the S3 logs myself. My logic has been to download all the files with the prefix, ie 2009-03-31. Then combine logs into one file. Then grep for a particular filename. The number of results is always about 20% less than if I grep for the same filename in your processed log file (for a specific date).Is this the way s3stat produces this processed log file? Is there something that I could missing that would account for the lower amount of results in my combined log file compared with S3 stat.We want to continue using S3Stat, but we just want to verify the numbers are correct. Any ideas?
Sean Fannan
Thursday, March 26, 2009
[ reply to this topic ]
[ return to topic list ]
|