Admin - what now?

comment by Silver (U6112)
posted 3 minutes ago
comment by Admin1 (U1)
posted 7 hours, 51 minutes ago
Drop a complaint when it happens again and I'll see if I can see what's triggering it.
----------------------------------------------------------------------
Might be a thicko but how do I raise a complaint if the site is down?
----------------------------------------------------------------------
Was it down or brutally slow?

It’s the Chinese.

I'll change the full backup to run at 2.30am, see if that's what's causing it.

comment by Admin1 (U1)
posted 24 minutes ago
comment by Silver (U6112)
posted 3 minutes ago
comment by Admin1 (U1)
posted 7 hours, 51 minutes ago
Drop a complaint when it happens again and I'll see if I can see what's triggering it.
----------------------------------------------------------------------
Might be a thicko but how do I raise a complaint if the site is down?
----------------------------------------------------------------------
Was it down or brutally slow?
----------------------------------------------------------------------
Good point! Not getting an immediate 404 but waiting for minutes before saying it is unavailable. When it did come back it was running like a dog.

Does the backup take an hour as that is the time it seemed to be a problem? It defo started around 21:55.

comment by Admin1 (U1)
posted 15 minutes ago
I'll change the full backup to run at 2.30am, see if that's what's causing it.
----------------------------------------------------------------------
Pray for gersfanoz

comment by Terminator1 (U1863)
posted 22 minutes ago
It’s the Chinese.
----------------------------------------------------------------------
The Scottish boards probably keep them well busy trying to interpret wtf we are talking about!

comment by Admin1 (U1)
posted 1 hour, 4 minutes ago
comment by Silver (U6112)
posted 3 minutes ago
comment by Admin1 (U1)
posted 7 hours, 51 minutes ago
Drop a complaint when it happens again and I'll see if I can see what's triggering it.
----------------------------------------------------------------------
Might be a thicko but how do I raise a complaint if the site is down?
----------------------------------------------------------------------
Was it down or brutally slow?
----------------------------------------------------------------------
It was down according to isitsownrightnow

comment by Silver (U6112)
posted 1 hour, 20 minutes ago
comment by Admin1 (U1)
posted 24 minutes ago
comment by Silver (U6112)
posted 3 minutes ago
comment by Admin1 (U1)
posted 7 hours, 51 minutes ago
Drop a complaint when it happens again and I'll see if I can see what's triggering it.
----------------------------------------------------------------------
Might be a thicko but how do I raise a complaint if the site is down?
----------------------------------------------------------------------
Was it down or brutally slow?
----------------------------------------------------------------------
Good point! Not getting an immediate 404 but waiting for minutes before saying it is unavailable. When it did come back it was running like a dog.

Does the backup take an hour as that is the time it seemed to be a problem? It defo started around 21:55.
----------------------------------------------------------------------
I'll need to look in to it. It might just be that we have hit a tipping point on a database index or something that is causing the tables to be scanned

comment by Silver (U6112)
posted 1 hour, 38 minutes ago
comment by Terminator1 (U1863)
posted 22 minutes ago
It’s the Chinese.
----------------------------------------------------------------------
The Scottish boards probably keep them well busy trying to interpret wtf we are talking about!
----------------------------------------------------------------------
at the moment, dubai usually

So judging by comment timestamps the site didn't go down completely for everyone but was brutally unusable, 9.53pm till 9.58pm not a single comment got through. Then it was one comment every few minutes.

The backup is scheduled for 10pm(now moved to 3am) so I think it's possibly unrelated to that and more related to arsenal match fishing at 9.54. The most computationally involved page on the site is the my membership page and those 5 comments, that show the last 5 commented articles that you have commented on. I've modified this query to limit that page such it will only show articles that have been commented on in last 30 days as opposed to all time. For frequent users the behaviour is identical but the computational overhead reduces massively, is we aren't having to get the article ids of every single article a person has made a comment on then check to see when those articles were last commented.

I'm hoping that buys us a fix, but it's informed speculation at the moment. It may be the indexes of the database are having to get flushed out to disk due to the size of site. Our indexes on comments table use about 4GB of ram alone.