12 thoughts on “Is it me or …”

  1. I have looked at the system log and HTTP server error log for the time around the report on the Bug Reports group and there were no suspicious messages there.  I did notice that “top” reported available memory around 1000000 which is low compared to where it usually runs, around twice that level.  “top -o %MEM” shows, as I suspected, that the memory is mostly used by a large number (32) of php-fpm pool worker processes.  This is evidence of a request flood like the ones we’ve seen before when rogue Web crawlers bombard the site with many requests per second.  When this happens, the pool of worker processes is exhausted and requests from other users have to wait in line until worker processes become available.  This makes the site slow to respond.

    I investigated the HTTP server access_log around the time of the report and saw no unusual Web crawler activity.  However, there was a storm of requests from IP address 220.14.234.25, used by some character going by the moniker 10Cents, amounting to 173 requests in 13 seconds, or around 13 hits per second during the period beginning at 2018-08-02T01:51:10.  A second storm of similar magnitude started at 2018-08-02T02:06:39.  The first storm was almost entirely requests to the image upload directory and may have resulted from the user, in administrator mode, displaying the media library, which shows all images uploaded by all users.  (Non-administrators see only their own uploads.)

    This request storm, whatever the cause, was more than sufficient to tie up the pool of worker processes and make the site slow down for everybody else, and is probably the cause of the reported slow response.  Assuming a storm is an isolated incident, response will return to normal once the queue of requests is cleared and worker processes become available.

    This vulnerability of PHP-FPM-based sites (which is the contemporary standard configuration for WordPress) is fundamental and baked into the design, and there is no IP or user based throttling to prevent an attacker from slowing a site’s response to molasses simply by hammering in a dozen requests per second for a period of several seconds.  The only defence against such abuse is to use a front-end service such as Cloudflare which provides denial of service attack mitigation and a content delivery network to offload requests for static content such as images to a content delivery network which caches content in edge locations, offloading requests from the site’s main server.  This, of course, costs money—their business plan, which would be appropriate for a site like this, is US$200/month, which would double the cost of operating the site and add one more possible point of failure and thing to administer.

    4+
    avataravataravataravatar
  2. John Walker:
    I have looked at the system log and HTTP server error log for the time around the report on the Bug Reports group and there were no suspicious messages there.  I did notice that “top” reported available memory around 1000000 which is low compared to where it usually runs, around twice that level.  “top -o %MEM” shows, as I suspected, that the memory is mostly used by a large number (32) of php-fpm pool worker processes.  This is evidence of a request flood like the ones we’ve seen before when rogue Web crawlers bombard the site with many requests per second.  When this happens, the pool of worker processes is exhausted and requests from other users have to wait in line until worker processes become available.  This makes the site slow to respond.

    I investigated the HTTP server access_log around the time of the report and saw no unusual Web crawler activity.  However, there was a storm of requests from IP address 220.14.234.25, used by some character going by the moniker 10Cents, amounting to 173 requests in 13 seconds, or around 13 hits per second during the period beginning at 2018-08-02T01:51:10.  A second storm of similar magnitude started at 2018-08-02T02:06:39.  The first storm was almost entirely requests to the image upload directory and may have resulted in the user, in administrator mode, displaying the media library, which shows all images uploaded by all users.  (Non-administrators see only their own uploads.)

    This request storm, whatever the cause, was more than sufficient to tie up the pool of worker processes and make the site slow down for everybody else, and is probably the cause of the reported slow response.  Assuming a storm is an isolated incident, response will return to normal once the queue of requests is cleared and worker processes become available.

    This vulnerability of PHP-FPM-based sites (which is the contemporary standard configuration for WordPress) is fundamental and baked into the design, and there is no IP or user based throttling to prevent an attacker from slowing a site’s response to molasses simply by hammering in a dozen requests per second for a period of several seconds.  The only defence against such abuse is to use a front-end service such as Cloudflare which provides denial of service attack mitigation and a content delivery network to offload requests for static content such as images to a content delivery network which caches content in edge location, offloading requests from the site’s main server.  This, of course, costs money—their business plan, which would be appropriate for a site like this, is US$200/month, which would double the cost of operating the site and add one more possible point of failure and thing to administer.

    Oops!

    Do you think it was from accessing the image library in Admin mode?

    1+
    avatar
  3. 10 Cents:
    Do you think it was from accessing the image library in Admin mode?

    That’s my guess.  I can’t think of anything else which would have generated that many requests for images from many different posts.  The image requests were not broken by requests for any other pages, just image after image after image.

    1+
    avatar
  4. John Walker:

    10 Cents:
    Do you think it was from accessing the image library in Admin mode?

    That’s my guess.  I can’t think of anything else which would have generated that many requests for images from many different posts.  The image requests were not broken by requests for any other pages, just image after image after image.

    Thanks you. I will use Dime more to be on the safe side.

    0

  5. Dime:
    I will use Dime more to be on the safe side.

    That is the best policy.  I only use my administrator account for things where it is absolutely necessary.  This is same policy Unix system administrators use with super-user (root) privilege; it is so easy to wreck things with a fat-finger that you only use root when there’s no other way to get the job done.

    3+
    avataravataravatar
  6. 10 Cents:

    John Walker:
    I have looked at the system log and HTTP server error log for the time around the report on the Bug Reports group and there were no suspicious messages there.  I did notice that “top” reported available memory around 1000000 which is low compared to where it usually runs, around twice that level.  “top -o %MEM” shows, as I suspected, that the memory is mostly used by a large number (32) of php-fpm pool worker processes.  This is evidence of a request flood like the ones we’ve seen before when rogue Web crawlers bombard the site with many requests per second.  When this happens, the pool of worker processes is exhausted and requests from other users have to wait in line until worker processes become available.  This makes the site slow to respond.

    I investigated the HTTP server access_log around the time of the report and saw no unusual Web crawler activity.  However, there was a storm of requests from IP address 220.14.234.25, used by some character going by the moniker 10Cents, amounting to 173 requests in 13 seconds, or around 13 hits per second during the period beginning at 2018-08-02T01:51:10.  A second storm of similar magnitude started at 2018-08-02T02:06:39.  The first storm was almost entirely requests to the image upload directory and may have resulted in the user, in administrator mode, displaying the media library, which shows all images uploaded by all users.  (Non-administrators see only their own uploads.)

    This request storm, whatever the cause, was more than sufficient to tie up the pool of worker processes and make the site slow down for everybody else, and is probably the cause of the reported slow response.  Assuming a storm is an isolated incident, response will return to normal once the queue of requests is cleared and worker processes become available.

    This vulnerability of PHP-FPM-based sites (which is the contemporary standard configuration for WordPress) is fundamental and baked into the design, and there is no IP or user based throttling to prevent an attacker from slowing a site’s response to molasses simply by hammering in a dozen requests per second for a period of several seconds.  The only defence against such abuse is to use a front-end service such as Cloudflare which provides denial of service attack mitigation and a content delivery network to offload requests for static content such as images to a content delivery network which caches content in edge location, offloading requests from the site’s main server.  This, of course, costs money—their business plan, which would be appropriate for a site like this, is US$200/month, which would double the cost of operating the site and add one more possible point of failure and thing to administer.

    Oops!

    Do you think it was from accessing the image library in Admin mode?

    Geez ten pennies, you’d think it was a porn site.

    0

  7. Ace Fungo:

    10 Cents:

    John Walker:
    I have looked at the system log and HTTP server error log for the time around the report on the Bug Reports group and there were no suspicious messages there.  I did notice that “top” reported available memory around 1000000 which is low compared to where it usually runs, around twice that level.  “top -o %MEM” shows, as I suspected, that the memory is mostly used by a large number (32) of php-fpm pool worker processes.  This is evidence of a request flood like the ones we’ve seen before when rogue Web crawlers bombard the site with many requests per second.  When this happens, the pool of worker processes is exhausted and requests from other users have to wait in line until worker processes become available.  This makes the site slow to respond.

    I investigated the HTTP server access_log around the time of the report and saw no unusual Web crawler activity.  However, there was a storm of requests from IP address 220.14.234.25, used by some character going by the moniker 10Cents, amounting to 173 requests in 13 seconds, or around 13 hits per second during the period beginning at 2018-08-02T01:51:10.  A second storm of similar magnitude started at 2018-08-02T02:06:39.  The first storm was almost entirely requests to the image upload directory and may have resulted in the user, in administrator mode, displaying the media library, which shows all images uploaded by all users.  (Non-administrators see only their own uploads.)

    This request storm, whatever the cause, was more than sufficient to tie up the pool of worker processes and make the site slow down for everybody else, and is probably the cause of the reported slow response.  Assuming a storm is an isolated incident, response will return to normal once the queue of requests is cleared and worker processes become available.

    This vulnerability of PHP-FPM-based sites (which is the contemporary standard configuration for WordPress) is fundamental and baked into the design, and there is no IP or user based throttling to prevent an attacker from slowing a site’s response to molasses simply by hammering in a dozen requests per second for a period of several seconds.  The only defence against such abuse is to use a front-end service such as Cloudflare which provides denial of service attack mitigation and a content delivery network to offload requests for static content such as images to a content delivery network which caches content in edge location, offloading requests from the site’s main server.  This, of course, costs money—their business plan, which would be appropriate for a site like this, is US$200/month, which would double the cost of operating the site and add one more possible point of failure and thing to administer.

    Oops!

    Do you think it was from accessing the image library in Admin mode?

    Geez ten pennies, you’d think it was a porn site.

    You could have saved me a lot of time by telling me this earlier. I just figured a name like Ratburger had to be hiding something.

    0

Leave a Reply