VIEŠIEJI FORUMAI/BENDRASIS FORUMAS

Temos pavadinimas: Server Downtime (Paprasta tema)

Autorius: System Administrator

Tema pradėta: 00:24:40 2010 08 31

Pranešimai: 10 Paskutinis pranešimas: 12:36:53 2010 09 01. Autorius EL

System Administrator Paskelbta: 00:24:40 2010 08 31

Pranešimai: 48

Temos: 50

Valstybė: United States



Has passed. I'm exhausted. Been sleeping far too little these past few days - a rundown of what happened therefore.

I have SkyLords set to replicate to my desktop computer, just incase anything should fail on the hard disk, I have backups to the very moment it failed, if I am booted into Linux. If I am booted into Windows mode, those updates are saved. I have to reboot weekly into Windows mode on my desktop computer however, what was the weekly downtime during the (fortunately briefish) time SL was hosted from my desktop - nobody was very fond of that solution, and it's permanent home is much nicer anyway, without pesky things like graphics cards and speakers, monitors and mice.

Well, as is amazingly common, we were under an attack from the internet on Thursday, some dumb cracker trying to guess our passwords, 10 per second or the like. Relatively boring stuff - but it was filling up our log files with failed login attempts. What ended up happening was a combination of factors which should never occur again. On Thursday morning, the disk filled up. This caused the white screen that many of you saw for a few days while I tried to free up disk space.

However, there was a lot of things, cron jobs mostly, waiting to happen. These sucked up disk space as fast as I could reclaim it, until finally I had most of the jobs pushed through - however, there were three jobs, written during my absence, I'll leave it to you to imagine how I feel about the author, that didn't fail gracefully - they continued to suck up disk space in an infinite loop. I found and fixed these final points on Friday night(?).

I wasn't content however to just leave it at that - I wanted to ensure that this triple disk hogging couldn't ever use up all of the 7 gb allocated... Far, far, too little, but I didn't think about that when I set it up back when. So, I tried to resize the disk. Again and again, I tried various tools. Finally giving up, I opened the case, pulled the disk, and resized it from my desktop machine :P.

When I put it back in however... It didn't boot. This was Saturday night around 3 am, and I needed to hit church the next day (was excellent however). Since I decided to let it rest for a day, I got back to it this evening after work (I was late to work already this morning, not able to work on the server). Now, past midnight, with less than 24 hours till the end of the round, I have fixed the booting problem (it was a strange kernel update that broke compatibility with SATA for some motherboards - I was unlucky to have the update from the brief time it wasn't fixed).

SkyLords now is running off of a ... much larger partition on a borrowed disk, and should be fine there for the foreseeable future. Just a note - before the downtime, the SkyLords server was running fine for almost a full 70 days without a single interruption. No reboots, hiccups, or needing to restart apache. SkyLords is running on some pretty awesome hardware, with a very wide Internet connection, and I dare you to NOT try to disrupt it please thank you very much :]. We are safe from any more accidental DOS attacks now however :P..

Thank you for your patience.

~Iiridayn


Omnibus Paskelbta: 02:17:08 2010 08 31

Pranešimai: 30

Temos: 10

Lytis: Vyras



Thanks Iiri!!!


TallDude3rd Paskelbta: 03:12:27 2010 08 31

Pranešimai: 1850

Temos: 37

Valstybė: Netherlands

Lytis: Vyras



Bruteforce attack on our server? That is just stupid




__________________________
UNOFFICIAL BIG KAHUNA OF THE UNIVERSE

EL Paskelbta: 03:25:42 2010 08 31

Pranešimai: 1705

Temos: 182

Valstybė: United Kingdom

Lytis: Vyras



So glad you got it fixed Iiri thank you :)

Yes it it pretty stupid attacking the server but at least we are now prepared for any future attacks


Tom Bomadial Paskelbta: 05:11:13 2010 08 31

Pranešimai: 423

Temos: 76

Valstybė: United States

Lytis: Vyras



Thank you.
Not only for fixing the system.
But also for going to church.




__________________________
"Is not easily provoked"
Translated:
It is not good to wake a sleeping bear

TallDude3rd Paskelbta: 06:10:00 2010 08 31

Pranešimai: 1850

Temos: 37

Valstybė: Netherlands

Lytis: Vyras



He is just going to church to program a text-speech so the preacher-man can take a break when reading the bible... lol




__________________________
UNOFFICIAL BIG KAHUNA OF THE UNIVERSE

Zerohours Paskelbta: 10:04:37 2010 08 31

Pranešimai: 2443

Temos: 287

Valstybė: Canada

Lytis: Vyras



TY Iiri




__________________________
I am here to help in anyway I can! Let's have fun!

Pray Paskelbta: 17:02:34 2010 08 31

Pranešimai: 2255

Temos: 147

Valstybė: United States

Lytis: Vyras



LOL I-Flight for sermons. Extreme & Deep.




__________________________
A storm is coming, a storm is coming in..

20:48:19 | Professor planet 0 from paladin
20:20:21 | Arcanix planet 14 Arcanix 01

Diplomacy & Trading Forum Moderator.



me naam is m Paskelbta: 04:10:17 2010 09 01

Pranešimai: 985

Temos: 37

Valstybė: Netherlands

Lytis: Vyras



*lol @ TD+Pray*

Anyway, I am wondering what or who was trying to ddos?




__________________________
Where has the rum gone!?

FORTRANshadow Paskelbta: 08:36:19 2010 09 01

Pranešimai: 383

Temos: 17

Valstybė: United States

Lytis: Vyras



Thanks, Iiri! I know how hard it is to keep things going in the big bad internet. Please know that we appreciate your efforts.

You may want to look at installing denyhosts on your server. Although it is normally configured just for ssh, I think the scripts can be set to review all login attempts in your auth.log.


EL Paskelbta: 12:36:53 2010 09 01

Pranešimai: 1705

Temos: 182

Valstybė: United Kingdom

Lytis: Vyras



It would work.


Ankstesnis 1  Sekantis

VIEŠIEJI FORUMAI
TICKETS
KALBŲ FORUMAI
SKYLORDS POKALBIAI