Well - that did something  . ...



200 = good request
302 = redirection
403 = forbidden
404 = page not found (hardly any of those)

We do appear (last 24 hours) to have had an over-enthusiastic search engine - it has made 7,000 requests since 03:30 this morning and I have asked it to cut the rate.  I have also asked a visitor from Tehran who's looking at ticket flows every 2 seconds and has called up 1000 so far to desist! We'll see how it goes

 
There appears to be another DDOS (distributed denial of service) attack underway - some silly spikes on our server load over recent days. I doubt that it's specifically aimed at us but it has given our server a lot more to do that usual and resulted in some sluggish responses to members.

Here is "now" ... with up to 1,000 requests per minute versus under a hundred in a normal minute ... requests arriving in flotillas from various IP addresses ....



Here is what it should look like



Edit to add - I have put in a filter to help identify the requests but won't be able to confirm its operation until the next flotilla of requests arrives.  The requests do not identify themselves as being from a spider or crawler and have a whole range of headers and as such appear to be designed to sneak under the radar.

 

I have seen the message about in some extreme testing I have done, but had no reports of it occurring for members to the extent that it irritates.  In any case in clears if you retry a few minutes later.

More "sillies" over the past few days, now fixed - sorry if the Coffee Shop had a blocked hot water nozzle for you earlier this week and was spluttering.



It appears that this was not a new problem - simply that a couple of the ongoing denial of service requests interfered with each other and confused our triage nurse code. I have put her straight.

 

I think we're down to around 15% of that

If you're genuinely here and see this occasionally - please let me know as it would mean I have overdone the hard shell protecting us

 
I noticed the other day we clocked up the most ever "visitors" at the same time on the 6th August with 1936 on line.   Shame when things like this skew the real picture.

 
Blimey!  There are apparently Luddites even worse than me, then!   

Full support to you, grahame, to do whatever is necessary. 

 
The server load is getting silly again with so many guests it's ridiculous ... and there's a lack of commonality in them such that there is no quick and easy solution to identify and block those that are clearly not genuine.  I do have a cunning plan - however it will require some experimentation and may result in some breaks of service and funny (peculiar) results for genuine users from time to time while I tune.   Please don't panic for an hour or two.

 
During July, traffic on our web server rocketed. On 30th June there were 263,933 accesses and on 31st July that was up to 597,214.  And the extras were ones that take significant server resource - requests that carry on sessions (my technical choice of measure) up from 93,049 to 492,532.  Our server IS being robust over this at present but there are some slow responses - members just please be aware.  Work underway to characterise the changes more finely and take any necessary follow up actions.

 
Here is where we were a week ago:


Here we are now:



And I'm much happier with how that has settled.

 

There used to be a pattern hour by hour with the server consistently three times as busy during the day than at night, and with weekend peaks about a half of those during the week. That was accounted for by it being primarily a server of specialist information for IT professionals.    It's a different server these days, of course - we're in the cloud and with a lot more compute power to handle requests, but the nature of those requests, and indeed what we serve, has changed.

If you see an address ending in ".html" it is highly unlikely it will actually be a file on a disc or in a memory somewhere that contains hypetext markup language - rather it will be a program that fills in a template with data from a database, sometimes with significant compute involved and with those databases having a long archive in them which influences the output.    The same applies to most of the ".jpg" images we serve.

Crawlers, bots and spiders - automata which for the most part are the same thing as each other - typically grab a ".html" page, look through it for hypertext links to other .html pages, go off and grab those, and so on until they have grabbed the whole public site.   We want them to do so, for the most part - at least the more useful content. Yes, I want to people to be offered Coffee Shop content when they search on Google, or ask Alexa ... I probably want plagiarism searches that the Universitys use to reveal that a student has copied from what a member wrote in 2016.  And I would like chatbots to be informed by our content too. 

But

* Compute power and storage is much cheaper these days, and so there are lots of spiders crawling around - almost an infestation

* The number of different pages (URLs) that we have ever grows with our long historic record

* The cost of servicing each individual request grows as each individual request needs to consider that individual record in the database.

* Extra facilities added

Combined, those three four are like cube quadratic rule, and the increases in data volume are phenomenal.  The image database which was set up because it was getting a bit big as folder with perhaps 500 pictures now serves some 20,000 using several gigabytes of storage.   Our little forum has over a quarter of a million messages on it to be searched through.   The new passenger flow system that I put in the other week has some 4 million records to analyse and sort and there are some 10,000 different pages for the spiders to find, each  of which "has to" to go through those records and sort the results.

Server load (just one of a number of monitors) from last night - ideally it should be at or below 1 job queuing at any time

The hard black line is yesterday ... the coloured lines are previous days

How do we control the load? Methods include:
1. We tell benign crawlers to avoid some areas on the server
2. We identify some crawlers and send them cached (slightly old) data rather than regenerating every time
3. We identify some crawlers and return a "go away"
4. Extra code added to quickly eliminate data in searches efficiently

 
I wonder - the big surge happened over a bank holiday weekend, and there's another coming up soon. The thought occurred to me after hearing a radio programme about the surge in the number of companies being registered at Companies House, using random addresses of unsuspecting British folk. The chap investigating this phenomenon found it was likely because of a recent ban, by their own government, on Chinese companies dealing in crypto currency. The traffic suddenly dropped for a few days, which he found were a public holiday in China. Could it be that the opposite works, and attempts at getting into networks rise when IT support staff are likelier to be fewer in number? Except at the Coffee Shop, of course.

 

That could be either a frustrating failure to deliver, or a security breach!

 
This reminds me of Humphrey Lyttleton explaining ‘one song to the tune of another’ on I'm Sorry I Haven’t a Clue: So imagine that the lyrics are the passengers, and the tunes are the trains. Now what happens if a train is accidentally shunted into the wrong siding..?

 

Thanks, TonyK and all the folks who have "like"d the thread.     

Monitoring / watching is ongoing with a flow of incoming requests and it's very much a question of being prepared to handle whatever hits us - and we can guess what that might be but are sometimes taken by surprise.

In many ways, there's a parallel with passengers arriving at a railway station.  Consider the Coffee Shop to be like Paddington for departing passengers.   

For the most part on the passenger side, it's pretty predictable how and when people will turn up and there are services they can board.  There may be occasional surges - for example from a news story / event, or when someone tells their friends what a marvellous service is offered.   Additional incoming infrastructure - the opening of the Elizabeth line - changes the flow of incoming passengers in ways in which the best we can do is try to predict the effect.

What sort of people will turn up?   They might be the nice, easy to handle passengers ... or they could have a disproportionate number with bicycles, heavy luggage, in wheelchair, or disproportionately needing the loo, all of which will put pressure on the station and service.

On the service side we have a number of platforms from which we can run services, a capacity of a number of trains, and also a number of tracks out on the line which have finite capacity, and each of these (disc space, memory and threads, and cpu resource) is finite.

Platforms, trains and tracks need some maintenance - not as much as the rail network, but things like a build up of rubbish needs clearing out from time to time, and we need to take copies of the setup so that if something goes wrong we have the tools to repair it.   And they also need monitoring and understanding so we are not taken by suprise when something does happen - either a fault or an overload, perhaps exacerbated by an out-of-pattern arrival of a lot of customers in a particular metric.  Lots of people with backpacks headed for Castle Cary ...

Where I flagged up a warning yesterday was that I was putting in various updates in the flow of how we get people onto trains and away, warning customers that there was a change I might accidentally turn some away.   We are still running within resources, but I was doing this as a proactive action to understand where the resources were / are going.

I have just started writing this and the parallels are surprisingly good - but there are differences.  If encouraged, I may continue to use the parallel for other explanations.

 
I came late to this party. I read and understood "There appears to be a very heavy server load at present", and got lost after that. I assume all is well again, and well done Graham!

 

Still thousands of requests per hour - our server received 314,169 requests in the 24 hours to 03:30 this morning which is a average of over 200 per minute.   I don't actually want to stop answering too many of these queries, but some of the answers need a lot of compute resource and we (or rather I) may be able to reduce that somewhat, especially for the "someone"s who may not need a full, current answer.

First steps to sorting the issues are to analyse them, and for that purpose I am putting in a restriction on certain accesses to see what difference it makes - I'll run it for a few hours, then perhaps move the restriction. If you should happen to get unexpected error pages that persist for more than a few minutes, please let me know.

 

Indeed - multiple people doing it, mind - and where there are multiple links in a page they will tend to go on and ask for lots of these follow ups in parallel.   "Nice" someones are a bit considerate ... others get selves labelled as "naught_boys" and that tends to land to restrictions or bans. 

 
Basically someone repeatedly knocking on the door, asking a question and without waiting for the answer asking another and another. 

 

One more than me! 

 
I've read all this and almost understood a couple of sentences!  Thanks to Grahame for keeping things on track for the rest of us to benefit.

 

Remains busy overnight with a high crawler / bot traffic; I have tuned a few settings but there's an element of that being a holding operation, and some of the tunings will take up to 24 hours to click into place.  I will take a further look in due course.

 

It would seem Claude is taking an interest in our public content ... 32,000 requests so far since this morning


https://www.anthropic.com/news/introducing-claude

Claude in not alone - there are lots of other crawler around too.  I put some advice to them in a "robots.txt" file nut have to know about them first.   There things are a double edged sword - in some ways they are helping themselves uninvited to our data resource, but then they help our visibility.  Sometime you ask Alexa and it will tell you an answer from the Coffee Shop.

---

I think the load is under control - I will watch overnight though.

 
One way to spend a Sunday evening.

The hidden side of running a forum. 

 
There appears to be a very heavy server load at present - I am investigating and think I know why - sorry about any sluggishness or broken connections in the next hour or so.