Server Fuckup - Socoder

-=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- (c) WidthPadding Industries 1987 0\|94\|0 -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=-
Socoder -> Off Topic -> Server Fuckup
Page : 1 Prev 2 Next
Sat, 04 Jul 2015, 23:55
Jayenkai	And, again, what the fuck is the point of me paying for a "Hosted Backup"? sigh I might scrap that in future. OK, the"Backup" is indeed now on the server, but annoyingly it's just been randomly plonked into a _restore/ path higgledy piggledy inside the server, thus requiring a lot of "copy this, copy that, ignore the other" type of fucking about. I was expecting it to be a complete smegging overwrite. GAH! sigh So, basically I'm paying extra for what amounts to a folder store, and nothing more. Fan-smegging-tastic. :\ ... still, at least the sidebar's not completely fudged, anymore. -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 00:31
Jayenkai	Wow, thanks Windows.. Yeah, absolutely.. That was DEFINITELY the best time to choose to do a forced restart.. sigh Right, I'm reuploading some of the lost files, and hopefully all should be back to some form of normality soon. Apologies again, and thanks for sticking with Socoder during this traumatic time!!! -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 02:51
Jayenkai	Although all appears fine, it'll be another day or so before everything's fully back to normal. Current Status : I'd rather check EVERYTHING, so I'm doing a complete website download to the Mac, where I'll triple-backup everything to my NAS and my Laptop, then I'll scan through it all with a fine toothed comb, and delete anything that shouldn't be there, before re-uploading everything that's currently missing from the server. (There's a whole bunch of stuff that I had to delete along the way!) Should be all good by tomorrow. Again, sorry this taking so feckin' long. I should've been keeping more backups myself. Serves me right for relying on the crapfest that is GoDaddy.. "Pay for Daily Backups!" seemed like such a good idea at the time.. :\ -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 04:13
Jayenkai	Can't believe I need to test things again.. Bah, humbug!! -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 05:14
Jayenkai	Ugh... That'll do for now.. I've just spent about 5 hours working on the bloody thing. Laptop needs charging, now. Quite a lot done, a LOT of tweaks made, but I've not yet started to delve into backups, instead manually doing a copy of the important stuff for the moment. .. in the meantime, the Mac is still downloading, so later tonight I should be able to re-upload any missing elements. Also, I'm completely aware that, somewhere along the line, I've buggered up the "Latest Items" list's memory. Going to have to sort that out. Hmmm... -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 06:29
rockford	It looks like everything is pretty much as it was (although I've not tested anything to any degree). So well done Jay. Time for a well earned rest. Any idea yet as to what did actually cause the problem? Or are the hosts not saying?
Sun, 05 Jul 2015, 08:45
Jayenkai	Here's the basic gist of what I think happened. A few weeks ago, I started to notice curious URLs around the net. The first of these brought to my attention was this one.. > Reveal 🔎 Twitter Linkage socoder.net/jstuff/canvasworld.de?topic=1463 That's a fairly fudged URL. And when I noticed it, I was intrigued, and found LOADS of versions all over the web. So, last week or two ago, I plugged that hole by creating a reload script when pages with URLs like that were opened. Somewhere a bug was in my script, and over the years a lot more bugs have also appeared. Error_Log enters the fray here. In general daily usage, the bugs that get recorded in error_log normally number ten or twenty. They're on OLD pages, like the Wednesday Workshop pages, and people rarely ever visit those pages. But all of a sudden hundreds of thousands of "links" were slightly different, and .. well, you know how Search SpiderBots are.. Page after page after page got reloaded within the space of a few days, some of which had silly old errors like "date() says that you shouldn't rely on the server time!" Most of those errors were fixed YEARS ago, but old stuff like the Workshop pages and .. Campfire? Remember that!? They were all being reloaded, all being rescanned, and each one threw up a tiny error. The error log defaults to a limit of 512Mb. So, as worrying as it was, it wasn't an issue that I had to cope with. After all, the Error Log counts towards your file-size, and the site still had a total file-size of 13Gb. (Socoder, all our uploads, wed workshops, and the entirety of AGameAWeek including all my games, screenshots, logos, icons, and even my complete music archive, all in the space of 13Gb!!) This continued to be a slight quirk, but "Not an issue" for a number of weeks. I kept heading in, using the error_logs to track down the quirks, and have actually fixed up an INSANE amount of the site over the past few weeks. (This is also why my own backup was now alarmingly out of date.. Backup, people!! And then backup again!!!!) All of these things culminated in the site riding on a wave of errors and fixes, and in all honesty, I do NOT believe that this was the cause of the problems. But.. Hey, you live and learn. If something's not broke, don't fix it. .. If it IS broke, get the fuck in there and fix it, ASAP!! When the Reddit thing happened a few days ago, I paid careful attention to everything. There was a LOT going on, and yet, with Spiders crawling all over the site, and Redditors poking through AGameAWeek, the server continued to cope surprisingly well. In fact, I'll be honest, I'm amazed it coped as well as it did. Our bandwidth usage for those few days is STAGGARING!! It's about 10 times what it normally is, and ramped to insane proportions. Luckily, we usually use about 0.3% of our monthly allowance, so we had oodles to spare. ... And so it continued. The error_logs recycled, the site coped well. On Thursday, something ELSE happened. I'm not 100% sure what changed and why, but given that Thursday was my "Ranting about Android" day, that's a bit of a co-incidence. In it's most basic form, the error_log stopped recycling, so the file grew and grew and grew. With Spiders still crawling all over the old outdated pages, the errors flooded in, and the error_log exploded. Quite why the server stopped recycling the error_log, I'm not sure. But that was the start of the issues, and everything else came from there. You can see it in the size of the backups. Before Thursday the backups were averaging 15Gb (Socoder, with "server" stuff added on!) On Thursday it shoots up to a 190Gb file! Eeek! There's not any one specific event in the error_log, it's just a whole sudden pile of shit all happening at once. Some of the errors look like they might be of the "PHP Updated and your code broke" type, but then if that was the case, why is this, old, backed up code not similarly broken.. :\ Hmm.. I dunno. I think somebody found a hole, and poked a stick. Something definitely happened that day, and I'm pretty sure it wasn't something that I did, nor was it one of the issues that were already cropping up in the Error_Log. shrugs Or maybe the server just went "FFS, Look, Errors!!" and REALLY wanted me to fix bits of Wednesday Workshop!? Anyway, we're hopefully through the worst. I spent the past few days plugging more holes than you could possibly imagine, and now have the error_log singing like a baby, even if you dare visit the Campfire. The error_logs are clean, and the site is running a little smoother. I've also disabled the rss feeds, because.. .. I'm not 100% sure what they were doing, but 230Mb rss feeds aren't normal! I need to look into that. .. And I'm definitely going to be keeping an eye on those logs from now on. -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 12:06
rockford	Wow! As you say, hopefully the holes have been patched and we won't suffer any more outages like that one. But if nothing else, it has helped you identify potential problems and stop them before they become actual problems in the future.
Sun, 05 Jul 2015, 12:33
Jayenkai	Yup, it's certainly fucked up! This never would've happened if we were still on a shared server -=-=- ''Load, Next List!''
Sun, 05 Jul 2015, 13:01
Dabz	Pleased she's sorted! Dabz
Sun, 05 Jul 2015, 17:05
cyangames	Gewhuhhhhhhh...shared...servers.... shivers <sarcasm>Just packing 1000's of websites on a server, what could possibly go wrong?</sarcasm> -=-=- Web / Game Dev, occasionally finishes off coding games also!
Tue, 07 Jul 2015, 01:18
Jayenkai	If you find a hole, don't poke the hole!! Let me know where it is, thanks! -=-=- ''Load, Next List!''
Tue, 07 Jul 2015, 01:52
cyangames	Will do
Page : 1 Prev 2 Next