Server settings changed - Stephen
Just to update you folk.

Looking right now at the server stats I see that it is basically being moderately hammered by the usage it is getting.
Even though RAM is not particularly short we hope to increase this on Saturday morning when it is a bit quieter.
The CPU is being kept quite busy and there isn't mcuh we can do about that straight away.
However, any slowdown - from ads or whatever - cause more connections to the database etc. and this was hitting the limit. Clearly old connections were also using up some of this allocation.
The upshot is that of about 10mins ago I have changed some of these settings which hopefully will improve things for the time being.
There are loads of things that can get tweaked. Hopefully I have tweaked the right ones.

Let's hopefully see some improvement from now.
----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - Stargazer {P}
Stephen,

Much better than an hour ago

thanks

Ian L.
Server settings changed - Stephen
... well as I am watchiing this thing closely now I notice it is currently real slow. Well that's because it is only 1% idle right at the moment as someone is checking on the site usage statistics on this site's admin section. This is a particularly tough query for the server as it involves searching some 3million records several times. i.e. it punishes the server for a while.

When this is done we shall see what it is really like ....



----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - Stephen
... actually the stats are being checked 4 times at the moment...

so the server is currently really being caned.

HJ, Mark, Dave - if the reply to the stats page is slow you need to be patient, because clicking the submit button again will only set the server off on another query while the old one sorts itself out. It normally takes around 30 seconds. If you click too many times it could take minutes and everyone will suffer.

Delete this when you are happy ...

S
----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - Stephen
You probably noticed that the site came to a complete halt just now.

There is a fault on the session logger database that I am currently repairing. I tried earlier, but it basically stops this site dead when that table is being repaired as every page accesses it.

I have temporarily turned off the logger while I repair it so that you can carry on browsing.

It should be fairly quick now - agreed?

S

----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - M.M
Quickest it has ever been Stephen...leave the air conditioning off all the time I say!

M.M
Server settings changed - Dan J
See - I told y'all the main man would find it in the end.

Seems better now Stephen, fingers crossed it lasts!
Server settings changed - M.M
Quickest it has ever been while viewing Stephen...leave the air conditioning off all the time I say!

Posting very slow though.

M.M
Server settings changed - Stephen
Ok this is now fixed.

The link logger is also now fixed but being checked.
Checking all other tables...

S
----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - Stephen
OK all are well apart from posts

You're not going to see a lot while this gets fixed.

S
----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - Stephen
Well I thought I could fix it without taking the forum down but within minutes there were so many people coming into the site that the server was in danger of queuing too many people and bringing the whole lot down.

Anyway this is all fixed now.

It should be quicker to post too.

Let's see
----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - Dynamic Dave
Stephen, from a moderating point of view things do seem a lot quicker. I can now juggle things around and edit/delete as and where necessary again.

btw, it was I that was looking at the statistics earlier - sorry. The reason I was running 4 queries was because I had 4 browser windows open and wanted to compare stats over the past few months. However the attempt was unsucessful at the time. I re-ran the query about an hour ago and there was a vast improvement to the speed of the enquiries.

Whatever you've done, Thankyou. Lets hope the gremlins don't get in overnight and change the settings again ;o)
Server settings changed - Stephen
The reason I was running 4 queries was because
I had 4 browser windows open and wanted to compare stats
over the past few months. However the attempt was unsucessful at
the time. I re-ran the query about an hour ago and
there was a vast improvement to the speed of the enquiries.


That's perfectly fine. The queries should take around 20 seconds to process. The problem is that they were taking minutes when you were having a look as the data file table for the stats was corrupted. I took the logger off line for a while and repaired the table and so now all is fine. The table is rather large - a few hundred MB in size and over 3 million records - so I don't think it does too badly. It adds to this table for everyone who visits this site. At some stage we can clear all the old rubbish out which will help a lot.

A few other tables were corrupted too - notably the forum posts table. That's why I took the forum offline for a few mins to fix this. I tried while every one was here and that was just too painful to watch a whole load of queries backing up waiting for the table to become unlocked during which time of course no-one could access the forum anyway.

We shall have to see how it goes. Basically during busy times the server is near the limit of number of connections and processing power to deal with that number of connections. So anything that slows it down - like a corrupted table or so - or some slow external ads - will rapidly take it to the limit. The extra ram will help - but it really needs both ram and more grunt.
I am having a few probs with the new machines as they have completely new version of Linux kernel, Apache, MySQL etc. and some of the libraries and tools we use need to play ball with this lot.
There is also going to be a backup server which is synced every half hour or so. So quite a lot to configure.
They are v nice machines though... so I am looking forward to them going in too. I am not looking forward to migrating all the sites across though...
Any budding Unix sys admins (geeks) out there willing to give me a hand? It's not the kind of thing I do every day - so I am not as good at this sort of thing as someone who spends their whole time in a Linux server.
Just send me an email. ;-)
----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - teabelly
The problem could be partly down to mysql itself. It is not fantastic at coping with lots of multiple table updates and can corrupt things rather nastily. Perhaps moving to Postgres is a possibilty? I don't know how the backroom talks to the database but moving over to using PEAR (a database abstraction layer) would make it easier to move between database backends. I don't know how the backroom loads the server but perhaps another discussion forum type eg phpbb or something might prove to be easier.

I'm more of a solaris & oracle geek but I have some experience of messing around with linux as I use it at desktop level at home and have a couple at work for experimenting on the server side with. I might be interested in some moonlighting ;-)




teabelly
Server settings changed - eMBe {P}
>>> "... the data file table for the stats was corrupted. I took the logger off line for a while and repaired the table and so now all is fine. ....
A few other tables were corrupted too - notably the forum posts table. ....." >>>>

So was this the problem all along and nothing to so with the ads?

Was my suspicion "Is it remotely possible that the work Stephen did on 10 July has somehow inadvertently caused a fault somewhere that has caused the engine to seize up?" (see www.honestjohn.co.uk/forum/post/index.htm?f=2&t=14...9 ) correct after all?
Server settings changed - Stephen
Was my suspicion "Is it remotely possible that the work Stephen
did on 10 July has somehow inadvertently caused a fault somewhere
that has caused the engine to seize up?" (see www.honestjohn.co.uk/forum/post/index.htm?f=2&t=14...9 )
correct after all?


Well anything that strains the server - or causes the too many connections limit to be reached - I have subsequently learned seems to cause the db to corrupt - just as the previous poster said. The ads certainly also took the server up to the connection limit it seems.

It seems that MySQL is OK if it is not at the limit of number of connections. Clearly the work I did on this day I should have run as 'nice' to give the server more time to process you guys.

I didn't - which was prob a mistake - so yes, it would have taken up a whole load of CPU and would have lead to you guys not being processed quick enough - server limits being reached and then poss db corruption.

The main changes I have made are:
- to reduce the timeout on persistent connections so that the server does not make too many connections to the db - but rather discards any not used for more than a few mins. It seems to do much better when it has fewer overal connections than loads just hanging around waiting to be used or with pending queries.
- increased the number of thread caches available - the threads created was tending to run away and this is now under control.
- slightly increased memory for joins and a few other such like things.

To answer the other person's post - we use the phplib abstraction layer - so yes all the code could be switched to Postgres if we wanted to.
I don't know whether MySQL 4 has addressed some of the issues we have talked about. That is what the new system will use anyway.

----------------------------------
Stephen Khoo
www.khoosys.net
Server settings changed - smokie
or....from moi

22 July 23:06 "I don't know much about web hosting though so I could be talking out of my back end...presumably SQL or similar is involved somewhere, it's indexes in that kind of thing which I think might have gone awry. Or, as has been suggested, an internal limit being reached"

Still, a million people can hypothesise but it doesn't fix the problem. Let's hope it's sorted now, it is quick enough this morning.
 

Value my car