Activity

  • John Walker posted an update in the group Group logo of UpdatesUpdates 5 months ago

    2020 June 30

    Updated the Subscribe to Comments Reloaded plug-in to version
    200629.  This is a minor update which modifies 15 files and
    deletes 4 (all language translation files we don't use).  The
    major change is adding an option to suppress the comment
    subscription request box for users who are not logged in.  This
    isn't needed at our site since the subscription box appears in
    the comment form and we don't show the comment form unless the
    user is logged in.  There are a number of structural chanages
    getting rid of some uses of jQuery (hooray!--jQuery is an
    abomination which transforms what is actually a very nice
    language, JavaScript, into something as ugly as PHP) and coping
    with sites which load their JavaScript libraries in the page
    footer.  None of these should affect our site in any way.
    
    We have no local code in this plug-in.  After syntax checking
    everything, I applied the update kit and everything appears to
    be in order.
    
    At 15:31:45 UTC the server crashed hard.  It would not even
    respond to pings within the same AWS data centre.  I tried
    rebooting it from the AWS console and nothing happened.  I then
    proceeded to try to stop and restart the instance, which usually
    causes it to move to different physical hardware.  I entered the
    Stop command, and after five minutes nothing had happened (I've
    seen failures to stop on AWS before, although not recently).
    Next, I proceeded to try a "Forcibly stop" command, which did
    nothing and after around another three minutes I did another
    "Forcibly stop" (I don't know if this does anything).  After
    another few minutes it finally stopped at 15:46.  I then started
    the instance, and at 15:47:48 it was up and running normally and
    appears to be working properly.
    
    There is nothing in /var/log/messages which indicates what went
    wrong.
    
    After the system came back up, I noticed we were being hammered
    with xmlrpc exploits.  Investigating the /tmp/gardol_wp_run.txt
    revealed that the iptables firewall was not running, which led
    me to the following in /var/log/messages:
        Jun 30 15:47:36 ip-172-31-24-57 iptables.init: iptables: Applying firewall rules: iptables-restore v1.8.2 (legacy): Set dropv4 doesn't exist.
        Jun 30 15:47:36 ip-172-31-24-57 iptables.init: Error occurred at line: 6
        Jun 30 15:47:36 ip-172-31-24-57 iptables.init: Try `iptables-restore -h' or 'iptables-restore --help' for more information.
        Jun 30 15:47:36 ip-172-31-24-57 iptables.init: [FAILED]
        Jun 30 15:47:36 ip-172-31-24-57 systemd: iptables.service: main process exited, code=exited, status=1/FAILURE
        Jun 30 15:47:36 ip-172-31-24-57 systemd: Failed to start IPv4 firewall with iptables.
        Jun 30 15:47:36 ip-172-31-24-57 systemd: Unit iptables.service entered failed state.
        Jun 30 15:47:36 ip-172-31-24-57 systemd: iptables.service failed.
        Jun 30 15:47:37 ip-172-31-24-57 cloud-init: Cloud-init v. 19.3-2.amzn2 running 'init-local' at Tue, 30 Jun 2020 15:47:37 +0000. Up 4.60 seconds.
        Jun 30 15:47:37 ip-172-31-24-57 systemd: Started Initial cloud-init job (pre-networking).
        Jun 30 15:47:37 ip-172-31-24-57 systemd: Reached target Network (Pre).
        Jun 30 15:47:37 ip-172-31-24-57 systemd: Starting Network (Pre).
        Jun 30 15:47:37 ip-172-31-24-57 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
        Jun 30 15:47:37 ip-172-31-24-57 ip6tables.init: ip6tables: Applying firewall rules: ip6tables-restore v1.8.2 (legacy): Set dropv6 doesn't exist.
        Jun 30 15:47:37 ip-172-31-24-57 ip6tables.init: Error occurred at line: 6
        Jun 30 15:47:37 ip-172-31-24-57 ip6tables.init: Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
        Jun 30 15:47:37 ip-172-31-24-57 ip6tables.init: [FAILED]
        Jun 30 15:47:37 ip-172-31-24-57 systemd: ip6tables.service: main process exited, code=exited, status=1/FAILURE
        Jun 30 15:47:37 ip-172-31-24-57 systemd: Failed to start IPv6 firewall with ip6tables.
        Jun 30 15:47:37 ip-172-31-24-57 systemd: Unit ip6tables.service entered failed state.
    Well, after quite a bit of head scratching, it turns out that
    saving and restoring the iptables (and ip6tables) configuration
    does not re-create the ipset objects which are referenced in the
    saved configuration.  When I manually created them with:
        ipset create dropv4 hash:net
        ipset create dropv6 hash:net family inet6
    and then restarted:
        systemctl start ip[6]tables
    everything came up fine.  The only remaining problem was that
    Gardol was not muzzling some xmlrpc attackers because it only
    muzzles when the number of strikes equals the strike-out
    counter, but not when it's above (to avoid redundant entries
    in the block list in the case of race conditions).  I then
    restarted Gardol with:
        /server/init/gardol_wp stop
        /server/init/gardol_wp start
    and now everything appears to be back to normal.  I will have to
    research how best to restore the ipsets around a reboot.
    
    OK, the ipset configuration is supposed to be restored by the
    ipset service.  But:
        systemctl status ipset
            * ipset.service - IP sets for iptables
               Loaded: loaded (/usr/lib/systemd/system/ipset.service; enabled; vendor preset: disabled)
               Active: active (exited) since Tue 2020-06-30 15:47:36 UTC; 41min ago
              Process: 1967 ExecStart=/usr/libexec/ipset/ipset.start-stop start (code=exited, status=0/SUCCESS)
             Main PID: 1967 (code=exited, status=0/SUCCESS)
               CGroup: /system.slice/ipset.service
    
            Jun 30 15:47:36 ip-172-31-24-57.eu-central-1.compute.internal systemd[1]: Starting IP sets for iptables...
            Jun 30 15:47:36 ip-172-31-24-57.eu-central-1.compute.internal ipset.start-stop[1967]: Loaded with no configuration
            Jun 30 15:47:36 ip-172-31-24-57.eu-central-1.compute.internal systemd[1]: Started IP sets for iptables.
    says it ran, and ran before it tried to start the firewall.  I
    don't know what "Loaded with no configuration" is all about.
    
    This turns out to have been the first reboot since we installed
    ipset, so the ipset persistence across reboots had never been
    tested before.
    
    After examining the:
        /usr/libexec/ipset/ipset.start-stop
    shell script which handles the ipset start and stop commands, I
    think I've figured out what is going on.  The tutorial I used to
    set up ipset:
        https://wiki.archlinux.org/index.php/Ipset
    says to save the configuration in /etc/ipset.conf, but that
    isn't where the script looks for it.  Instead, it looks for
    /etc/sysconfig/ipset and if it is not present, reports "Loaded
    with no configuration" as we saw.
    
    I modified our ~/bin/Garback script to save to
    /etc/sysconfig/ipset which will enable the automatic restore of
    ipset parameters.
    
    Modified /etc/sysconfig/ipset-config to set:
        IPSET_SAVE_ON_STOP="yes"
    which will save the ipset configuration on a normal shutdown.
    Note that this configuration lies about where the configuration
    is saved, naming yet another red herring destination.
    
    Committed the Subscribe to Comments Reloaded plug-in version
    200629 update (Build 480) and published the changes to GitHub.