Re: [Paddlewise] PaddleWise list censorship

[ At the risk of diverting into email policy, but since multiple
people have brought it up...I'm going to try to cover all points
in one message instead of sending multiple followups.  I think if
those goes any further, we should take it off-list.  ---Rsk ]

Nitpicking point of order:

Paddlewise isn't a listserve.  It's a mailing list.  Listserves are a subset
of mailing lists, and are only those which are run by ListServ software.

The rest of use run our lists with Mailman (currently best available
option), majordomo, listproc, ezmlm, or other software.  So they're
just "mailing lists", that is, there's no special terminology for them.

Captchas:

Captchas are (a) a very bad idea and (b) largely ineffective.  [Some]
spammers pay appallingly low wages (by western standards) to people in
third-world countries to sit in cybercafes (or equivalent) and solve
them for 15 hours a day.  Or they solve them programmatically, and given
the increasing sophistication of those programs, we have already passed
the point where "captcha elaborate enough to evade automatic decoding"
is not "captcha easy enough that a typical human being can decode it".
This is the death knell for captchas.

Not to mention that they don't work for the visually-impaired or for
people who aren't using GUI interfaces.  Like me, right now, and most
of the time, in fact.

Captchas earned their own section in the anti-spam book I'm writing; they're
included in the chapter on ill-conceived and/or obsolete techniques. ;-)

Anti-spam:

If you're really getting that much spam, then the problem is quite likely
your perimeter defenses.  (See below for discussion of the DROP list.)
We here at RockAndWater run a number of paddling-related mailing lists
(and are happy to host more), and we use a layered approach to anti-spam,
beginning with: the firewall.  Particularly egregious spammers get
their packets dropped before they get anywhere near a mail server.
We follow that up with aggressive filtering at the SMTP level, using
DNSBLs, local blacklists (of networks, hosts, subdomains, domains, users,
etc.) and we enforce mail server sanity checks (e.g., forward and reverse
DNS must exist).  And on the lists themselves, we enforce members-only:
anything else that makes it that far gets queued for moderator attention.

We typically see one message a week that makes it that far, and almost
all of those are messages from list-members who are sending from an
address other than the one they're subscribed at.  Mailman (which is
what we use) makes it easy to deal with those.  In the cases where it
really *is* spam (a) we discard it and (b) we often add it to the
appropriate place in the anti-spam config so that we won't be bothered
by that particular source again.

We *do not* use content filters per se.  Content filters (a) perform
poorly (b) require lots of resources and (c) really aren't necessary
given all the other stuff we're doing.  And while they were marginally
effective for a while some years ago, spammers have long since evolved
very sophisticated methods for fooling them.   It's become a battle
wherein increasingly large resources are being expended (on our side)
for rapidly decreasing reward.  Moreover, content filters often trip
over unobtrusive content, a simple example of which is "discussion
about spam" vs. "spam".

So -- and this will surprise some of you -- we reject almost all the spam
that we reject *before we even see the body of the message*.  We don't
need to wait for it; we know what's coming.  So we issue a reject notice
and hang up the connection, so to speak.  (Think of it this way: if the
last 10,000 times we heard from the host, it was trying to deliver spam...
then on the 10,001st, it's probably trying to deliver spam.)

So -- slightly repeating myself -- if you're getting THAT much spam,
then most likely it's not being aggressively rejected upstream.  If you're
running a paddling resource, and you'd like me to review your configuration,
I can do that, provided you have access to the necessary info about it.
No charge: I consider it good river karma to help out where possible.
(Non-paddling resource?  Let's discuss my exorbitant and entirely
unreasonable consulting rates.)

Step 1.0 that you should have already done:

If you run a mail server, or mailing list, or web-based forum, or ANYTHING,
then you want to go here:

	http://www.spamhaus.org/DROP/

and download the Spamhaus DROP list.  DROP stands for "Don't Route Or Peer",
which is network-geek-speak for "don't talk to or listen to these networks".
The DROP list contains networks that are 100% spammer-owned-and-operated,
or hjiacked, or both.  Not only do you not want email from these chunks
of network space, you don't want ANY traffic at all.  Nor do you want
to send them any.  Nothing good (for you) will come of it.

"You will never find a more wretched hive of scum and villainry."
Yeah, that's what the DROP list covers.

The DROP list is best used in your perimeter routers and/or firewalls.
It comes in CIDR format (and if you're network-clueful, you know what
that is).  The drill is: download it once a month, install in your
devices along with the moral equivalent of "drop all packets to/from",
and enjoy the very nice reduction in malicious traffic of all kinds.

If you don't control the network your resource is on, then you might
be able to implement the drop list in the application you're running,
whether it's SMTP or HTTP or whatnot.  The DROP list easily integrates
with most sane mail systems and web servers.

Side-effects of spam:

At least one person commented along these lines:

"I'm not getting mail that I want because my mail provider is blocking it."

This isn't uncommon.  However, don't blame your own ISP without a full
understanding.  It may well be the case -- and it often is -- that the
source of "mail you want" is precisely the same as the source of "a lot
of spam".  You see, some unscrupulous ISPs/web hosts will sell services
to (a) legitimate customers and (b) spammers and then deliberately locate
them in the same network block, adjacent to each other, like this:

	[...]
	192.168.0.23	niceperson1.com
	192.168.0.24	spammer1.com
	192.168.0.25	niceperson2.org 
	192.168.0.26	spammer2.com
	[...]

Particularly unscrupulous ISPs/web hosts will actually swap the IP
addresses of legit/spamming customers or pull other tricks designed
to render blocking techniques less effective.  This is called "using
a human shield" and it's designed to blackmail others into accepting
their traffic by generating pressure from customers who are upset about
"not getting mail they want".  (As well as pressure from niceperson1.com,
who are often totally unaware that they're being used in this fashion.)

So don't be surprised if the reason your ISP is rejecting the handful
of messages you want is that they're coming down the same pipe (or "tube"
if you're former Senator Ted Stevens) as thousands of spams per hour.
It's an ugly business.

Competence:

One of the unfortunate things that's happened this decade is that a
frighteningly large number of mail system admins who have little clue
about email and even less about spam have decided to "solve" their
problem by deploying some very poorly designed and built anti-spam
systems.  Many of these "solve" the problem by redirecting it, which
is the email equivalent of cleaning up the trash in your yard by
picking a random neighbor's and throwing it in theirs.  (Google for
"backscatter" for an explanation of this.)  They've been aided and
abetted in this by any number of vendors which have sprung into
existence in order to make a quick buck from the Internet's collective
misery.  Some of these vendors charge rather a lot of coin to provide
some really bad systems.  (Ours is entirely built using open-source tools
and data sources, and is far better than any offering from any vendor.)

Anyway, among the numerous downsides of this is the difficulty in
convincing said mail system admins that they're doing something
wrong.  "It works for me", they will often reply, and yes, just as much
as throwing their trash in someone else's yard does, "it works for them".
It has actually proven necessary to (a) publicly name-and-shame some
of them and/or (b) blacklist them, in order to direct their attention
to the fact that they're selfishly making the problem worse.  For example:

	http://www.backscatterer.org/

which (as of 11:22 PM 23 Apr 2009) lists 439260 individual IP addresses
that are busily engaged in making our lives worse by doing the
throw-your-trash-at-someone-else trick.  And those are just the ones
seen within the last four weeks, and "seen" by the people running
that particular list -- so it's a tip-of-the-tip-of-the-iceberg.

Finally:

Keep in mind that SMTP (which is the protocol that moves mail around)
is *designed* to be "best effort" and no better.  It performs incredibly
well despite that, and despite the fact that mail servers are under
24x7 assault by a LOT of attackers, some of whom are very clueful,
and some of whom have a lot of resources to throw at them.  So my
advice is that we should all be very cautious about pointing fingers
at anybody about any non-delivery/delayed delivery of email messages.
A lot of the time, the explanation is benign.

---Rsk
***************************************************************************
PaddleWise Paddling Mailing List - Any opinions or suggestions expressed
here are solely those of the writer(s). You must assume the entire
responsibility for reliance upon them. All postings copyright the author.
Submissions:     PaddleWise_at_PaddleWise.net
Subscriptions:   PaddleWise-request_at_PaddleWise.net
Website:         http://www.paddlewise.net/
***************************************************************************