2019 May 11
Over time, the automatic firewalling of IP addresses from which login storm attacks have originated has resulted in a very long list of banned addresses: as of this writing a total of 2459 single IP addresses are banned. Many of these addresses are probably floating IP addresses which were co-opted into botnet attacks and have since been reassigned to other people. In many cases, we haven't seen a single dropped packet from an address for a long time. Since we now have the /server/log/gardol/gardol_wp_log file which logs all automatic blocks by gardol_wp, it is possible to clean up the block list by purging older IP addresses which have not offended recently. I wrote a new program: /server/bin/gardol_wp/gargol.pl which analyses the block list and the gardol_wp_log file and prepares a list of candidates for un-blocking based upon the following criteria. 1. Only single IP addresses are candidates. When an IP range is blocked, that indicates manual banning of a rogue network. 2. Only addresses with no dropped packets since the last reboot are candidates. 3. If an entry for the IP address appears in the $gardolLog file, compute the time since the most recent block of that address. If the time is greater than or equal to $unbanTime, the address is a candidate. 4. If no entry for the IP address appears in the $gardolLog, the address is a candidate for un-banning if it is among the first $unbanCountN entries in the IPTABLES drop list (entries are listed from oldest to most recent). This program is intended to be run after the system has been up for a long time (say, 30 days or more), and before the block list is saved prior to a reboot. When run, it prepares a shell script, candidates.sh, in the current directory, which contains iptables or ip6tables (depending on the address type) commands to remove the candidate from the INPUT tcp DROP list. This script must be run manually as super-user to actually un-ban the candidate addresses. The rule for un-banning addresses which do not appear in the gardol_wp_log is intended to handle addresses which have been manually banned for some reason. We use their order of appearance in the block list as a proxy for their age. Note that since the dropped packet statistics are not preserved across reboots, it only makes sense to run this utility when the system has been up long enough to provide a sample of activity from IP addresses. The fact that it has been, say, 90 days, since an IP address was blocked according to the log does not imply it hasn't sent a packet over that time: just that it hasn't sent one since the last reboot. Keep this in mind when setting the parameters for un-blocking. The parameters are currently set to: Un-ban based on age: 60 days or more Un-ban based on index: IPv4: first 100, IPv6: first 20 After testing this in dry run mode, I proceeded as follows. Performed a: service iptables save service ip6tables save to preserve the firewall rule changes before pruning the block list. This saved the rules as follows: 3819 /etc/sysconfig/iptables 459 /etc/sysconfig/ip6tables Made a mirror backup to Juno. Made a backup AMI: Ratburger Backup 2019-05-11 ami-07f6ca92871798975 / snap-06d317f8dd6177aa2 /server snap-08faa74937770c415 Prepared the un-block candidates list: cd /server/bin/gardol_wp perl gargol.pl which reported: IPv4: 2186 blocked IPv6: 272 blocked Gardol_wp log: 4163 unique entries loaded. 1036 candidates written to candidates.sh. Backed up the candidates list as candidates.sh_2019-05-11 just in case we want to review whether some offenders come back subsequently. OK, big gulp--apply the un-block list. Note that if something horrible happens we can always restore the tables saved before this step. super sh -x candidates.sh After this step, we count the number of blocked sites: muzzle | wc -l 3231 Looks good. Now re-ran gargol.pl to see how it evaluates the situation. It now finds just 6 candidates, presumably sites which just went over the age threshold. Installed all pending update packages: 32 in total, 19 for security including a new kernel, HTTPD, openssl, and PHP. Backed up /etc/sysconfig/iptables and ip6tables in /server/bin/gardol_wp just in case. Saved the now-pruned iptables block list to preserve across the reboot. service iptables save service ip6tables save Now the length of these tables are: 2908 /etc/sysconfig/iptables 336 /etc/sysconfig/ip6tables Rebooted at 14:50 UTC. The system had been up for 41 days. The system came up normally after the reboot. We are now running on kernel 4.14.114-103.97.amzn2.x86_64. After the reboot, both the IPv4 and IPv6 firewall rules were restored automatically.