Outgoing Spam Monitoring
The SpamExperts filters are extremely effective at blocking a large percentage of outgoing spam/viruses, to prevent issues with your network reputation. However, it is essential that you are proactive in stopping the abuse at its source by suspending any spamming customers/accounts. If such accounts are not suspended/blocked, there may eventually be a spam run which is missed by our engines. You can prevent any such spam escalations (or other type of attacks from abusive customer accounts), by ensuring the account is locked down before it starts to cause real issues. Our systems allow you to quickly and easily identify such abusive accounts, before any third-party issues occur.
There are a number of ways that spammers can be monitored via our systems.
Control Panel Quarantine (Local Cloud Only)
If you have super administrator access to the control panel, you can review the blocked outgoing spam emails on the "Spam quarantine" page in the Outgoing section as Super Administrator. Although the number of daily spam emails you find there can be overwhelming at start, spending 15 minutes a day to analyze/block the source of the most frequent messages you can find there will quickly result in a significant drop of overall spam traffic.
This is strongly recommended when starting to use our filtering, so your administrators can easily pinpoint the top spam causes and get more familiar with tracking down/blocking the spam sources.
Best Practice for Smarthost Users
- Ensure all your smarthost authentication users are grouped as part of a single administrative domain (e.g. out.yourcompany.tld)
- Configure your sending MTA to always include an end-user identification header
- Set your outgoing user account to use this identity header
- Manually/automatically locate abusive identities and shutdown the main spam source (and temporarily lock down the identity via our identity management as an immediate measure)
Managing Outgoing Spam
Outgoing Log Search
You can view outgoing blocked messages from the Admin, Domain or Email Level Control Panel using the Outgoing Log Search:
- Select Outgoing > Logs
- In the Query Rules panel, filter using Status > is one of > Quarantined
- Click Show Results to list all matches
Outgoing Reports page
You can view senders/Identities in grouped format using the Outgoing Reports feature from the Admin or Domain Level or Control Panel.
- Select Reporting > Outgoing Reports
- Select the relevant domain if accessing from the Admin Level
- Select the Period
- In Classification, select Rejected (or Accepted if you wish to see accepted emails and not quarantined ones)
- In the Group by dropdown, select identity
- Click Show to display all results
Classifications
A list of Message Classifications can be found by clicking on Classifications in the description at the top of the Log Search page.
Classifications relate to the message Sub Classes that can be found by adding the Sub Class and Extra Class columns to the view using the Columns to be displayed dropdown.
More information on what classifications mean and how these can be dealt with can be found here:
Automatic and Manual Locking
Manually Lock Identity from the Outgoing Reports Page
- Click the lock icon next to the identity
- To unlock the identity, click the lock icon again
Automatic Locking based on the Identity header within the Control Panel
You can choose to auto-lock senders based on their Identity header. For this to work, there must first be a configured Identity.
To start autolocking senders based on this you need to make sure the option Lock Identities Automatically:" is set to "Yes" in the outgoing user settings page:
- Select Outgoing > Manage users
- Locate the outgoing user you want to configure, and from the dropdown, select Edit. The Outgoing user settings page is displayed
- Ensure the Lock identities automatically option is set to Yes:
Identities will be locked when a certain amount of spam, phishing or virsues is seen in a short time frame.
The locked identities can continue to be seen via the log search and outgoing reports page.
Manually Lock Identity from Outgoing Log Search
You can choose to lock a sender based on their identity header from this page:
- Locate the relevant message and select Lock Identity from the dropdown:
- In the prompt, enter a reason for locking this sender and click Confirm
You can also lock the Outgoing user from here (by selecting Lock user in the dropdown). This would prevent any outgoing mail being sent from that outgoing user (IP or domain).
Manual locking from the Manage Identities page
The manage Identities page gives you a comprehensive overview of the current locked and unlocked identities. To lock or unlock either on mass or individual identities you can do the following:
- Use the drop down menu next to the Identity
- Click lock
- Enter a reason for the lock
- Click Execute
To unlock, do the reverse.
Manual Locking senders based on the identity header from the API
When using IP authentication, it's often needed to be able to lock specific senders without locking the whole IP. This can be done by locking senders via their Identification method (Envelope-sender, Authenticating User, Identification Header). To do this you need to execute the following API:
https://apihostname/cgi-bin/api?call=api_lock_outgoing_identity&domain=DOMAIN&identity=bob@example.com&username=USERNAME
To list the current locked users:
https://apihostname/cgi-bin/api?call=api_list_locked_identities&domain=DOMAIN&username=USERNAME
Please note, to be able to use this method, an Identification header must first be set as mentioned above.
Alternative reporting
Outgoing Spam Reports via CSV
There is an option to have a daily CSV report for outgoing spam per outgoing user. The API call to activate the daily report with spamming accounts is: api_set_outgoing_report_recipient(domain, recipient='', username='') > "".
Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.
https://MASTERHOSTNAME/cgi-bin/api?call=api_set_outgoing_report_recipient&domain=DOMAIN&recipient=RECIPIENT&username=USERNAME"
So this should be good for you in regards to the monitoring of outgoing spam, and be able to see overall information. (please note , subject and body will not be shown here)
Alternative Outgoing Spam Reports via CSV based on "Identification header"
There is another option to have a CSV report for outgoing spam per identification header sender that is sent every 2 hours to a specified email address. The API call to activate the daily report with spamming accounts is:
api_set_outgoing_report2_recipient(domain, recipient='', username='') > "".
Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.
https://master.hostname/cgi-bin/api?call=api_set_outgoing_report2_recipient&domain=DOMAIN&username=USERNAME&recipient=RECIPEINT
The report will contain counts of blocked spam per Identification and counts of invalid senders.
For example:
"Authentication Domain","Authentication User","User Identification","Spam Count","Invalid Sender Count" example.com,,bob@example.com,100,0 example.net,,example.net,235,301 example.org,,example.org,0,2000
IMAP quarantine access
Rather than using any of the scripts, or the Spampanel webinterface, you can authenticate with your "global" administrator account (Local Cloud only) using any IMAP compatible email client for real-time access to the spam quarantine. Please contact our support in case you do not have the "global" credentials yet.
Global quarantine reporting script
Rather than using Spampanel or direct IMAP quarantine access to review the quarantine, this is a script that will parse the outgoing IMAP quarantine.
(Local Cloud Only) You can download this here: http://download.seinternal.com/tools/retrieve_quarantine_info.py
Usage: retrieve_quarantine_info.py [options] Output a list of quarantined outgoing messages. Options: -h, --help show this help message and exit -c, --csv saves output to csv file -d, --display displays the loglines as they pass by -i, --incoming search the incoming quarantine -o, --outgoing search the outgoing quarantine -t, --today load results from today (otherwise yesterday) -s IMAPHOST, --imaphost=IMAPHOST The hostname of the imap server -u IMAPUSER, --username=IMAPUSER The username to check, usually 'global' -p IMAPPASS, --password=IMAPPASS The password for the 'global' user -n, --no-bounce filters out mail originating from 'mailer-daemon'
Please make sure you run this from a NON FILTERING server only. This will retrieve the quarantined messages either in an on screen format or saved to a .csv file. For example you could do something like this: $./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME
This will display on screen the messages that have been quarantined outgoing in the last hour. You will be prompted for a "Global" password. This is given out only on request. If this is required please contact support@spamexperts.com for more details.
This will show you something like this:
$./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME (or quarantine server if applicable) Please enter the password for the IMAP account 'global'. Password: #,From,To,Reply-To,Qmail UID,Invoked for,IP/Username,Evidence,PHP script,Auth.sender,Auth-User,Auth-Email 1,--DATA WILL BE HERE-- 2,--DATA WILL BE HERE-- 3,--DATA WILL BE HERE-- 4,--DATA WILL BE HERE-- 5,--DATA WILL BE HERE--
Here you can see the information on the blocked messages and some relevant details. Alternatively it may be easier for you not to display the data and save it to a csv file, then you can open it in any excel like program and sort it on the specific field to group the data. This should then give you a better idea of some of the clients that you can close down for spamming.
Using the Log Search API
It's possible to use the api_find_outgoing_messages to be able to get for example a list of the top 50 spammers in X amount of time. A bash example can be seen below:
curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&classification=oversize%2Cblacklisted%2Clocked%2Cphish%2Cvirus%2Cspam%2Cdeferred&include_in_progress=False&id=&subject_header=&api_language=en&columns=sender" | sort | uniq -c | sort -nr | head -50
It's also then possible to start automating actions. For example, you can use 2 API's 'api_find_outgoing_messages' and 'api_blacklist_outgoing_sender' to be able to query the log API to find senders that have sent X amount of messages in X amount of time, and then take a further action. The example below shows a way to check the outgoing logs for the last 24 hours, and if the sender has sent more than 2000 messages then block the sender.
curl -s -k "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&include_in_progress=False&columns=sender" | sort | uniq -c | grep "@" | sort | awk '{if($1==$1+0 && $1>2000)print $2}' | xargs -I{} curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_blacklist_outgoing_sender&domain=DOMAIN&sender={}'
While these are very basic examples, our API is very versatile, so these can be amended or changed to suit your exact needs, for example, by locking specific senders based on the identification headers or more.
ARF reports
An ARF report is sent each time an outgoing spam message is blocked, and will contain a copy of the original message including headers. For information on how to set this up, see Configure the Abuse Report Address.
Many larger companies already process ARF reports originating from external sources such as AOL. You can set your administrator address to point to your existing ARF parsing infrastructure, so your existing abuse handling systems automatically receive and process our datafeeds.
ARF parser
If you do not have an ARF parser yet, we recommend that you set up a system to handle your incoming ARF reports. We can recommend the free opensource software Abuse.IO for this. Alternatively you can e.g. use a python file that can parse the contents of the ARF reports. Your sysadmins will know how best they can utilize this and parse the data that they need.
Using ARF automation also allows you to accept ARF feed from third-parties, to further improve your abuse handling and to deal with abuse that does not (yet) use our outgoing filter.