Spam Prevention BCP
Spam Prevention BCP
5 September 2006
APNIC 22, Kaohsiung, Taiwan
1
Presenters
• Champika Wijayatunga, APNIC
• Kazu Yamamoto, IIJ
• James Lick
2
Overview
• Session 1 & 2
– Background: spam
– Problems and prevention
• Consumers, Businesses and ISPs
– Spam filtering
– Handling spam
– Spam laws
– APNIC involvement
• Session 3
– Anti spam technologies
– Port 25 blocking
– Domain authentication
• Session 4
– Case studies
3
Background - spam
4
Quick quiz! :-)
6
Statistics – how critical?
• Nearly 75% of email traffic is spam
– Over 1 billion unsolicited messages sent per month
– Amount is doubling every 5 months
• AOL & Hotmail block around 2 billion spam each
day & still more slipping through
– Now the figure is 10 times higher than that of 5 years
ago
Source: http://www.postini.com/stats
7
Statistics – how critical?
• Spam volume grows at 37% per month
– an annual growth of 400%
• Lots of spam appears to use foreign relay
– Countries may need to work on spam
legislations
• Court cases between spammers & innocent
victims
– Only major corporations can afford such court
cases
Businesses
Consumers
ISPs
Problem
Annoyance
Pornography Severe Problem
Lost productivity
Server strain
Bounce messages
Dictionary attacks
Complaints
Support costs
Spoofing
Bulk messages
Bandwidth costs
9
Source: Competitive Enterprise Institute
Problems & Prevention:
Consumers
10
Problems for consumers
• Privacy
11
Email validation process
• Spammers are interested in only active
accounts
– Not only valid address but also active ones
12
Email validation process
• Spammer can determine validity based
on the response
– Ex: “This account does not exist”,
“Account could not be found” , “The
recipients inbox is full” etc.
13
Email validation process
• By sending a series of messages,
attackers can determine
– What time of day the user reads email
– How often the user checks mail
– What email program user uses
– What operating system is being used
– Whether user uses HTML or plain text
email
– Whether user always use the same
computer to check mail etc.
14
Prevention
• Use caution when choosing sites
• Avoid giveaways & other “too good to be
true” sites
• Avoid signing up for sites that use an opt-
out policy
• Read sign-up screens carefully
• Read privacy statement carefully
15
Prevention
• Know where your email can be found
• Guard your primary email address
• Never click reply to unknown senders
• Be careful with your browser
• Choose an ISP that actively blocks spam
• Find out how to filter your own email
16
Problems & Prevention:
Businesses
17
Problems for businesses
• Technical support costs
• Spoofing (use of legitimate name)
• Harvesting e-mail ids of staff
• Phishing attacks
• Sexual harassment
• Marketing difficulties
18
Web crawlers, robots
• Robots or spambots are used for email
harvesting
• These tools work like browsers and catalog
information found
– Robot makes a request for a particular URL
19
Web crawlers, robots
• The robot also performs tasks with HTML
on each page
– eg: count pages for statistical analysis, index
pages for search engines, mirror the content of
web pages, etc.
20
Web crawlers, robots
• This technology can be used to find and
extract email addresses
– email addresses follow a particular pattern or
regular expression (ex: “@” symbol)
21
Email patterns
• It can be easy for a spammer to guess
email patterns for most companies
– eg: first initial and last name are used to form
an email address
– A simple run through the alphabet with
common last names yields many valid hits
22
User exposure
• Friends
– Forwarding emails
– New users who haven’t faced bad experiences
may be less cautious than more seasoned
users
• Parsing of lists
• Address books
– Help these users
23
Tracking emails to gather
information
• Many scams and hoaxes
• HTML mail
– Email messages can contain colours, fonts
and embedded graphics
– Image isn’t actually sent but connects to the
website when the email program loads
• Web bugs
– Track the emails
– How many times the mail program access the
contents etc.
24
Hyperlinks
• Similar to web bugs
– But require some interaction from users
• Instead of simply viewing or opening an
email message, the user needs to click a
link or button
– So the spammer knows the email account is
active
• As with web bugs, hyperlinks can be coded
to indicate what user clicked the link
– The user may also be asked to supply additional
information
25
Vacation auto responders
• Spammer determines that the email
address is active
– More information can be retrieved (time of the
email message read, IP address, email
program etc)
– Some times the vacation responses can
provide more info for spammers
26
Vacation auto responders
Jeff
27
Vacation auto responders
Jeff
28
Spoofing email identities
•
Return-Path: <[email protected]>
Received: from [66.38.203.132] by e-hostzz.comIP with HTTP;
Sun, : 31:55 +0400
From: “Tim” <[email protected]>
To: [email protected]
Subject: Re: CYXS, Contact !
Mime-Version: 1.0
X-mailer: mPOP Web-Mail 2.19
X-Originating-IP: [e-hostzz.comIP]
Date : Sun, 16 Jun 2006 11:37:55 -0700
Reply-To: “Tim Wright” <[email protected]>
29
Phishing
• Starts as an email message to get users to
go to a web site
– To enter personal details for use in an identity
scam
30
Using email addresses for other
purposes
• Web applications routinely store email
address as data and as user ID
– Any vulnerability in a web application’s security
can reveal this sensitive information
– Need to use unique IDs
31
Error message reasoning
• Error messages from web applications can
expose email addresses
– Login pages, forgotten password, registration
are focus points for these type of attacks
– The attacker can keep trying email IDs until the
error message gives a clue
32
SQL injection
• Ex: web site that displays job listings can have a
link such as
– http://www.mycompany.com/Jobs.asp?id=6236
33
Prevention
• Robot exclusion standards
– Tags direct a robot to ignore the document and not
follow any hyperlinks contained on the page
• <META NAME=“ROBOTS” CONTENT=“NOINDEX, NOFOLLOW”>
34
Prevention
• Spam poison (sending fake email ids)
– Doesn’t prevent email addresses from being harvested
but attempt to taint the results
36
Prevention
• Be careful when using vacation auto
responders
– Restrict auto responder to certain people or
those that matches particular rules
• Challenge responses
– Server holds all email from unrecognized addresses
and sends an automated message
38
Problems & Prevention:
ISPs
39
Problems: Costs
40
Problems: Costs
• Bounce messages
– Spammers usually put a fake email address in
the Reply-To header to avoid bounces
– Another ISP or user ends up getting thousands
of bounce messages, clogging the servers
41
Problems: Costs
• Dictionary attacks
– Try multiple combinations of letters, at a
popular domain name
• This puts a huge drain on ISP’s servers
• Customer complaints
– Consumes a lot of helpdesk and customer
service time
– Large amounts of objectionable email can
drive customers away
42
SMTP
• SMTP is simple
– No mechanism to verify the sending
server or the accuracy of the from
address
– SMTP server has no way to verify
messages such as “This message is
from your bank and concerns your
account” etc.
43
Prevention
• Contractual and cooperative solutions
– Acceptable Use Policy (AUP)
• Spammers try to operate through open relays or by
hijacking ISPs other than their own
• ISPs need to have strong anti-spam policies
• Prohibit these customers from sending spam
through ISP servers
44
Prevention
45
Prevention
• Contractual and cooperative solutions contd
– E-stamps
• Sender agree to pay money per message if the
message is reported as spam
46
Prevention
• Software solutions
– Software can partially stop the spam problem
at several levels
• Efficient tools for end users to control spam
• Blocking techniques for ISPs
• Sender authentication programs
47
Prevention
• Whitelists
– Lists of servers known to be sending valid,
legitimate emails
– The address of the sending server can be
compared to a whitelist
• Blacklist filtering
– Opposite of whitelists; lists of servers known to
be operated by spammers
– Block all incoming mail from the blacklisted
addresses
• Many blacklists block all IP addresses from specific
countries
48
Prevention
• Multiparty solutions
– Need collaboration between ISPs, bulk
mailers, and consumers
• Options of redesigning the SMTP
- Probably based on security certificates
- Should be a secure, verified protocol like HTTPS
49
Prevention
• Force accountability by identifying who is
sending the message (e.g. spam, phishing
& viruses)
– Email authentication systems
• SPF (Sender Policy Framework or Sender Permitted
From)
• Caller ID
• Sender ID
50
Prevention
51
Prevention
• SPF only checks for spoofing at the
message transport level
– Verifying the "bounce back" address for an
email, which is sent before the body of a
message is received
– Tells the receiving email server where to send
rejection notices
• Caller ID
– Use sender authentication technology
• Tries to validate source address associated with
an email message
53
Prevention
• Caller ID contd.
– Email servers & clients that receive
messages check the DNS record
– Match the “From" address in the message
header to the published address of the
approved sending servers
– Email messages that don't match the
source address can be discarded
54
Prevention
56
Prevention
• Legal solutions
– Legislation that targets fraudulent or
destructive conduct
– Forged headers can be made illegal
– Illegal to send emails with falsified routing
information
– Labeling ( ex: [ADV:] or [ADV:ADLT] )
– Mandatory unsubscribe or opt-out (options to
reject emails) requirements
– Restrictions on email harvesting
– Opt-in (options to receive email)
57
Spam filtering & anti-spam services
58
Provider-based
mail filtering Black hole &
services signature / rule
subscription services
Internet
Sendmail /
Boarder router
MS Exchange Web server
email server Firewall proxy
DMZ
Gateway-based
Server based Internal router mail filtering
spam control appliance
Internal
Network
Client-based
spam control
59
Mailbox filtering in email programs
• Use mail folders
– Spam can go into the trash folder
60
Using filters
61
Spammer’s tricks to evade filters
• Capitalisation
– eg: filter may look for spammersrus.com but
spammers can use SpammersRus.com
• No text
– Many spam messages contain only graphical image of
text
• Wrods Speled w.r.o.n.g
• Hidden bogus codes
– Ex: instead of make money it says ma<m>ke
mon<n>ey
– Filter can get confused with the HTML tags
62
Server-side spam filtering
63
Server-side spam filtering
65
Server-side filtering techniques : Bulk
counting
• One of the most effective approaches
– These filters look at incoming mail to try to
recognize when many similar messages are
arriving
• Each time a message arrives, the filter makes a
hash (compressed) code representing the contents
of the message
• Looks in the database to see how many other
messages arrived recently with the same hash
code
• If it’s several, the message is probably a spam
66
Server-side filtering techniques: Bulk
counting
• Spammers tend to change their
messages to avoid bulk counting
filters
• Effective bulk counting filters should
have “fuzzy” hash codes
– Designed to disregard minor differences
between one copy of the message and
another
• Any bulk counting system needs to be
configured to whitelists
67
Server-side filtering techniques: Bulk
counting
• Bulk counting doesn’t need to be restricted
to a single mail server
– Can exchange hash code information among
many servers
68
Server-side filtering techniques: Timing
and greylists
• Timing techniques and greylists
– Filters can often detect spam by looking
at peculiarities of the rate at which it
arrives
• Body filters
– These filters look at the contents of spam
– As the server filter can see all incoming
mail, Bayesian and other adaptive
techniques can use a larger sample base
69
Server-side filtering techniques: Timing
and greylists
• Most spam is sent by spamware
• As there are no error checkings, viruses
and worms can get away
• These spamware and viruses can be
detected by looking for timing peculiarities
caused by the lack of error checking
70
Server-side filtering techniques: Timing
and greylists
• During mail exchange, the sequence of
commands & status messages are
predictable for successful message delivery
– Spamware sends all the commands without
waiting for replies
71
Server-side filtering techniques: Timing
and greylists
• A mail server can be short of disk space or
other problem that temporarily keep it from
receiving mail
– It returns temporary error status codes
• Real mail programs retry the message
• But spamware and viruses don’t bother
72
Server-side filtering techniques: Timing
and greylists
• With greylisting when a server sees an incoming
message from an unknown server:
– The server returns a temporary rejection message and
keep track of the IP addresses
73
Server-side filtering techniques: Timing
and greylists
• This process might create delays
– Rejects nearly all mail sent by spamware
74
Server-side filtering techniques:
Combination filtering
• Sequentially filtering
– Apply multiple tests sequentially
– Do the IP tests first as the remote host
connects
• Then the bulk tests
• And the body tests
– If any of the tests identify a message as
spam, the filter stops and doesn’t do any
more testing on that message
75
Server-side filtering techniques :
Combination filtering
• Scoring filters
– Run all their tests, assign a weight to
each test and add the weights of the
tests that the message passed
– If the score is above a threshold level,
the message is considered to be spam
• Sequential filters can be much faster
because they often don’t need to run
the full set of tests
– But harder to tune than scoring filters
76
Filtering on UNIX/LINUX servers
• Most of the email software and filtering add-ons
for UNIX are open source or freeware
• Most widely used UNIX mail server is sendmail
– Provisions to plug in many mail filters with direct
support for DNSBLs and a milter (mail filter)
77
Filtering on UNIX/LINUX servers
• UNIX/Linux mail servers also use procmail
filtering package
– Procmail has its own pattern matching
language
• Most popular UNIX/Linux filter is
SpamAssassin
– www.spamassassin.org
– Can use DNSBLs, DCC etc along with fixed,
heuristic, and Bayesian filters
78
Anti-spam programs
• Most of the email programs may not have
truly effective spam filters
79
Anti-spam services
• Spam filtering is a complex and CPU-intensive
application
• Better to dedicate a separate server
• Many vendors offer anti-spam devices
– Already configured with anti-spam software that
logically sits between the Internet and the existing mail
server
• Network mail configuration is adjusted
– Incoming mail goes to the appliance which examines
the mails
– Then re-emails the filtered mail to the existing mail
server
80
Checklist for server spam filters
82
Email headers
Return-Path: [email protected]
Received: from ns.isoutsider.com (unknown
[210.109.171.2]) by receiving.my-isp.com
(8.9.3/8.9.3) with ESMTP id FSW930923; Sun, 31 Aug
2003 22:59:28 -700 (PDT)
Received: from adventures (CPE –
65-31-127-1.wi.rr.com [65.31.127.1]) by
ns.ioutsider.com (8.11.6/8.11.6) with ESMTP id
h7JFLKK09863; Sun, 31 Aug 2003 22:56:22 +0900
Message – Id:
[email protected]
Received: from billclinton.whitehouse.gov
([184.325.23.124]) by mailout.yahoo.com (Postfix)
With SMTP id 7600A32641; Sun, 31 Aug 2003 11:40:44
-0700 (PDT)
From: [email protected]
To: <undisclosed.Recipients>
Subject: Look Great for the Spring with Discounts
on HGH (human Growth hormone)!!!!
Date: Sat, 30 Aug 2003 02:10:21 -0800
MIME-Version: 1.0
Reply-To: [email protected]
Errors-To: [email protected]
83
Following the flow of email headers
84
Following the flow of email headers
• As you are sure that your ISP may not be
sending you spam, you can look for
ns.isoutsider.com
85
Following the flow of email headers
87
Investigating contents of spam
• Example
89
Sending complaints
• Nicely :-)
– Don’t transfer your anger at spammer to
the ISP
– Spamming isn’t really ISP’s fault
Dear Administrator,
Thanks!
90
Sending complaints
93
Fighting spam with spam
95
Spam laws
96
Characteristics of spam
• Solicited or unsolicited
– Was the message sent to someone who
specifically asked to receive it?
• Permission and relationship
– Did the recipient of the email address
give permission, either expressed or in
some sort of implied fashion?
• Commercial or noncommercial
– Does the email message advertise the
commercial availability of a product or
service offered for sale or lease? etc
97
Characteristics of spam
• Fraudulent content
– Is the email message advertising an illegal get-
rich scheme or a bogus work from home? etc
• Bogus opt-out
– Does the message offer to remove you from its
mailing list, but when you click the link the
removal web page doesn’t exist?
99
Spam laws
100
Spam laws
• Contact info
– email to contain the company’s name
and a physical address or other contact
information
• No using third party’s domain name
– Using anybody else’s domain name to
send spam without their permission
101
Spam laws - a comparison
102
Australian Spam Act
103
Messages covered by the Act
The Spam Act covers commercial electronic
messages that are sent by applications such
as:
– Email
– Short message service (SMS)
– Multimedia messages service (MMS)
– Instant messaging (iM)
104
What is considered as spam?
• Electronic messaging (emails, SMS,
etc.)
• Commercial in nature
• Unsolicited – sent without prior consent
• The Spam Act makes no reference to
bulk messaging
– A single unsolicited commercial electronic
message could be a spam
105
The penalties
• A business found to be in breach of the
Spam Act may be subjected to a penalty of
up to AU$220,000 for a single day’s
contraventions
• Repeated breaches may result in penalty of
up to AU$1.1 million
106
3 steps to ensure compliance
108
International laws
• Enforcement of penalties relating to spam
coming from overseas can be problematic
until international arrangements are in place
109
APNIC’s involvement
110
Detecting the spam/abuse
111
Detecting the spam/abuse
• If a standard search refers you to APNIC
– Means only that the network in question is
registered in the AP region
– Does not mean that APNIC is responsible or
that the hacker/spammer is using APNIC
network
112
Investigation of complaints
• APNIC is not able to investigate these
complaints
• Can use the APNIC Whois database to find
out where to take your complaint
• APNIC does not regulate the conduct of
Internet activity (legally or in practice)
113
Investigation of complaints
• Investigation possibilities
– Cooperation of the network administrators
– law enforcement agencies
• Local jurisdiction
• Jurisdiction where the problem originates
114
How can APNIC help you?
• The APNIC Whois Database
– Holds IP address records within the AP region
– Can use this database to track down the
source of the network abuse
– Can find contact details of the relevant network
administrators
• Not the individual users
• Use administrators log files to contact the individual
involved
115
How can APNIC help you?
• Education of network operators in the Asia
Pacific community
– Address policies and the importance of
registration of resources
• Community discussions can be raised in
the APNIC open policy meetings, mailing
lists, etc.
• Spam BOFs
116
Summary
• Background: spam
• Problems & prevention
– Consumers, Businesses, ISPs
• Spam filtering and anti spam techniques
• Handling spam
• Spam Laws
• APNIC involvement
117
Questions?
118