My main forum recently had an episode of server overloading which from log analysis seems to have been caused by an overly aggressive search engine bot which simultaneously opened several sessions from different IP addresses bombarding our site with page requests. Our site typically gets about 10 hits per second during a busy period, but I think this bot was sending something over 100 per second, for several minutes at a time, repeated several times per day.
This resulted in our server hitting its CPU resource limit and eventually mysql hit its queue limit and everything went a bit pear shaped. The site and server also became more or less unmanageable during the episode.
I tried enabling the phpBB load management feature, but it really made things (from our user's perspective) worse. It is slow to operate so the initial response is not improved, and even worse, it is slow to recover, so users lose out a both ends, and they don't like the forum locked message.
I think that it is worth considering the addition of a feature to limit server load caused by individual phpBB users. The scheme I am thinking of is:
- Limit per-user hits
- For each logged-in user, limit the rate of pages served per (~10s) interval, across all sessions for that user.
- For each bot limit the rate of pages served per (~10s) interval, across all sessions for that bot (maybe a different limit)
- For guest sessions similar but per IP address.
- Hits exceeding the governed rate are responded to with code 429 Too Many Requests (https://tools.ietf.org/html/rfc6585) this response will take much less server resource than a normal page response.
- New session creations are rate limited in a similar manner (to lower limits)
- The rate limits could be made dynamic reducing when the measure server load nears a critical level
- Probably get get sufficient benefit from 2 steps (normal and high load)
- Bots could be completely locked out under high load
- Misbehaving users and periods of high load would be logged.
My thought is that although this could (probably) be achieved as an extension, it should be added to the phpBB core.