Handling Abuse (and why Brazil is blocked)

Servers have a finite amount of resources, from RAM to CPU, and possibly have data caps as well. Then there is an even greater resource that has to be kept in mind, the allowance to continue hosting the server in it’s current location, or for the server to utilize other services hosted elsewhere.

Anything that causes RAM usage to escalate, or CPU usage to skyrocket, are abuses of the resource from the standpoint of the server’s owner. These could result in a DoS (Denial of Service) wherein whatever thing that is consuming excessive RAM may cause other services hosted to be unable to operate due to inability to access the required memory, or whatever thing is consuming CPU could cause itself, or other services, to be unable to operate, due to lack of time to schedule the processes involved. These are awful, but are capable of being remedied by the owner via a plethora of means. They cannot, however, compare to the other two resources previously defined.

If the use of a service at the server causes the server to abuse external systems, the best case scenario is that the server is now disallowed from utilizing those externally hosted services, but if the abuse is especially horrendous, the location hosting the server may elect to no longer allow the owner to host their server and it’s services at their location. The loss of the use of an external service is likely easily remedied by the owner, typically by demonstrating to the externally hosted service the issue was resolved, or by choosing a new, comparable, service. The loss of the location for hosting can render an owner into ruination. One might argue the owner can change the location no issue but sometimes, it means losing things that are not easy to regain. For instance, if the IP is tied to the location, they could lose the IP’s positive reputation for mail delivery (SMTP) and have to rebuild that reputation at a new address, and that’s if the new location allows the owner to run SMTP services.

In general, any owner wants to keep from having their server’s resources abused in any of the aforementioned fashions.

It should also be noted that one can abuse a service so as to cause it to generate a bunch of logs which could eventually fill disk. We’ll leave it to the reader to determine the risks associated with such as they are fairly obvious.

The same server hosting this blog, hosts a DNS/DoT server as well, and it was discovered that a bunch of queries were overly repeated. The abusers were repeatedly requesting the TXT record for cisco.com from the DNS server we host, from a multitude of IPs. We do not understand the purpose as to why they did so either.

There were over 30 million request over the past 30 days for this, and the top client (my home network) has only sent 1 million requests over the past 30 days. Sending 30x a legitimate user sends is pretty much abusive.

The worry here is that it might be being used in a DNS Amplification Attack (unlikely, the DNS server rate limits somewhat), or that because our DNS server was pestered for the result repeatedly it might be talking to resolvers upstream and pestering them with the same, which could result in our DNS server being rendered unusable if the upstreams determined our continued requests were abusive. This led to a need to resolve the issue, no matter if it was highly likely that negative consequences were to be experienced. With a focus on resolution, determining other statistics kind of falls by the wayside. Who cares when it started, what it started with, and what our resources have done or will do, the focus is on resolution which means focusing on making the abuse… stop.

As a little aside here, let’s consider the mind of an abuser to some minimal degree. Generally they think things like “I have so many IPs, they can try blocking me but they’ll never find them all”. In general, this might be correct. However, they rarely consider other tactics available to those mitigating the issue.

Going back to a mitigator’s mind, we now think on how we resolved the issue.

Our process goes like this:

1. Find the IP that is generating abusive traffic and block it.
   • All fixed? If yes, stop here, otherwise, continue.
2. Look for similarities in the IPs that are generating abusive traffic.
   1. All seem to be within a same subnet, so find the CIDR subnet involved and block the entire subnet.
      • All fixed? If yes, stop here, otherwise, continue.
   2. Determine the ASN hosting the IP’s involved in abuse traffic, and block the entire ASN.
      • All fixed? If yes, stop here, otherwise, continue.
   3. Determine if all the ASN’s share a similar country/region, and block the entire country/region.
      • We’ve yet to find something that fails at this step…

So we started with step 1, and blocked a single IP, but found it inadequate, and kept looking till we determined to start with that we saw all the most recent abusive traffic coming from ASN 262287 and ASN 396356. latitude.sh owns these ASNs (they were formerly MaxiHost). We proceed to block their ASNs and send them an abuse notification.

Then things got hairy, the abuse didn’t stop, so we started doing ASN blocks for IPs when we realized it was from a few in the same range, and the range was owned by a singular ASN.

This resulted in blocking the following ASNs:

266630

Fios Tecnologia

63956

5G Network Operations Pty Ltd

268747

L C S Wifi LTDA

263394

Networks Wireless Telecom

52769

Lagosnet Internet Banda Larga LTDA

263280

I4 Telecom LTDA Me

265420

Antonio Carlos Atella Ferreira Junior

263536

Microset Maquinas E Servicos LTDA

61642

Nexnett Brasil Telecom

264117

Provedor Fibramais LTDA

There were more ASNs but this is a large enough sample set for one to draw some conclusions.

Most of the ASNs listed above originate from Brazil with an example exception of 63956 and the latitude.sh ASNs, and the abuse was yet to end, so we finally escalated to blocking all the ASNs with Brazil origination’s.

We seriously exhausted our options, going through quite a few ASN blocks, before electing to block the entirety of Brazil, but after doing so many, it was determined that we could either play wack-a-mole, or sadly punish all of Brazil for the actions of a few and save ourselves some time and effort. I have doubts that the services of my server are used by many/any Brazillians and all I can say is… send me an email and I’ll look into removing your ASN from the blacklist.

With FreeBSD this is fairly effortless and if one is using PF, with firewall config reloads being atomic and the “last match wins” type policy, one can be exceedingly certain they won’t accidentally lock themself out.

Basically, you declare a persistent table, add a rule to block traffic coming from any IP/CIDR in that table, destined to an IP/Port combo of your selection.

The below shows basically what we used. It should be noted that we don’t take abuse lightly and if a single service is abused, you won’t be blocked from just that service, but from all services hosted here.

table <blacklist> persist

# normalization rules go here

# translation rules go here

# rules go here

# service blacklist goes below
block drop from <blacklist> to any

# permanent rules for anti-lockout go here

We then did the following to add each IP/CIDR in to the table:

pfctl -t blacklist -T add $IP_OR_CIDR

After having blocked all of Brazil, the abusive traffic stopped. It’s a love/hate resolution. Love that the abusive traffic is done, hate that we had to block all of Brazil to achieve it though. There’s no expiry for the rules either, so we’ve basically committed to never servicing Brazil again which might be a loss for us at some point perhaps. We may yet experience further unforseen consequences due to the abuse and it’s mitigation but cannot predict any negative consequences arising from this resolution.