Cleantalk extension for phpBB can remove spam posts, plus its spam firewall feature is very useful

This is an update on an earlier post on removing spam posts.

Removing spam posts is hard because it requires actually reading the post and deciding if the post is spam or not and then using moderator tools to remove these posts. If your forum is overwhelmed with spam posts, this is a Herculean endeavor. Ideally though posts could be “read” by software and it would make the judgment on whether it is spam or not.

The Cleantalk extension for phpBB 3.1.x and 3.2.x can do just this as well as lots of other really cool tricks. My customers love Cleantalk, but the service is not free. However, it is so inexpensive that it easily justifies spending $8/year for the service. You can subscribe on the Cleantalk website. As of this writing, you can try it for free for 7 days. After 7 days, it won’t bring down your forum but it will stop working.

What is Cleantalk?

Cleantalk is essentially a huge database of addresses of known spammer sites. While it’s not perfect, based on the experience of my clients it is about 99% perfect. I originally recommended it as a spam registration solution for my clients. It still does that but is less necessary since phpBB 3.2. This is because since phpBB 3.2, version 2 of Google’s reCaptcha is supported. Unless it gets hacked, as long as you have it properly configured as a spambot countermeasure it should prevent virtually all spam registrations.

However, it has two powerful features that still keep it relevant for phpBB forums.

Cleantalk ACP Interface
Cleantalk ACP Interface

Installing and enabling Cleantalk

Cleantalk is installed like any other extension. While it can be downloaded from phpbb.com, you should download it from Cleantalk instead or from its GitHub page. This is because as of this writing the version on phpbb.com does not include the spam firewall feature, and you will probably want to enable this feature. You can access it through the Administration Control Panel: ACP > Extensions > Antispam by Cleantalk. Before you can do much with it you have to enter your Cleantalk key which you can get from their website or by pressing the button in the extension that should retrieve it for you.

Removing spam users and spam posts

As you can see from the image, once the extension is enabled and the key is properly configured there is a prominent Check users for spam button on its page within the Administration Control Panel. If you have lots of users, it may hang. Based on my experience though the next time you go into its interface you will see a list of potential spammers.

As I said, it is not perfect. So I recommend that for users with posts to check these out these users topics to make sure their posts are spam before deleting them. For those you want to delete, check the boxes next to their usernames and then press Delete marked. You can also press Delete all to remove all users and their posts. You may have to go through many pages to delete all spam users and their posts, but this is obviously much faster than doing a visual inspection of all your posts.

Spam firewall

This is a new feature which as of this writing is not available if you download the extension from phpbb.com. It keeps almost all spammers from hitting your site at all. Instead, Cleantalk’s servers grab it first. In the event the user is legitimate, there is a link that will take them to your website.

Why is this useful? Because it reduces the stress on your server by limiting it to legitimate traffic only. It speeds up the performance of your forum and makes it less likely that you will have to pay for the cost of a higher class of hosting to handle your traffic. Isn’t that worth $8 a year?

Stopping contact form spam

Cleantalk has one other useful feature: the ability to stop contact form spam. Of course you can disable the contact form (ACP > General > Contact page settings) and that will solve that issue. Or you can have Cleantalk essentially moderate it for you, passing on only valid contact forms to you. Simply check that option on the extension’s page and submit the form. Somewhat oddly, the phpBB group did not tie the contact form to the spambot countermeasure feature of phpBB. Perhaps that will come in a future release.

In any event for forums that get lots of spam and/or lots of traffic, using the Cleantalk service with the Cleantalk extension for phpBB is a no-brainer providing you know about it. Now you do!

Fixing phpBB emailing problems

Note: this post was updated on February 10, 2019.

Having trouble sending out emails such as mass emails or email notifications using phpBB? If so, you are not alone. Sending mass emails is a feature of phpBB (ACP > System > General tasks > Mass email). Users can also sign up for various email notifications in the User Control Panel. For some boards though these email notifications just doesn’t seem to work reliably, and sometimes don’t work at all. Perhaps some but not all of these emails go out.

Even if you don’t send out mass email regularly, phpBB has a number of features that depend on outgoing email servers to send out emails in a timely manner to all intended recipients. It’s been my experience that solving these issues is hard. This is particularly hard on shared hosting where there are quotas on outgoing emails. It’s made more confusing by phpBB’s inelegant interface for dealing with emails.

phpBB hands off email, generally to your host’s email server. Generally you have no idea where this email server is, what its policies are and how to determine if the email server is actually sending the emails. It often requires a support ticket to your web host, who may or may not provide accurate information.

Sorting this out can be very confusing which is why you might want to hire me. In most cases though I will have to work with your web host to fix these problems. Here are some useful techniques that can often solve these issues.

Setting phpBB’s email settings correctly

You can find your forum’s email settings in the Administration Control Panel: ACP > General > Client communications > Email settings. Things to check:

  • Enable board-wide emails. This should be Yes. Obviously if set to No, no emails will go out.
  • SMTP settings > Use SMTP server for email.
    • This is normally set to No. PHP is configured to send emails through your web host’s email server. This makes emailing simple in that you don’t need to know anything about the configuration of the email server. However, sometimes this will bite you. Here’s a list of reasons why using SMTP instead might be better approach.
    • If you have Windows hosting, you usually have to set this to Yes. In addition you will have to configure the SMTP server information fields appropriately using information provided by your web host. Getting SMTP email configured correctly is often challenging so SMTP, so if you are not sure what your SMTP settings should be, ask your web host. In addition, some web hosts don’t allow their servers to connect with external SMTP email servers. If you see in phpBB’s error log (ACP > Maintenance > Forum logs > Error log) messages like “Could not connect to smtp host : 110 : Connection timed out” this is probably because your host won’t allow external SMTP connections. If they provide an internal SMTP server, that is usually allowed.
  • Email package size. This is the maximum number of emails that will go out in one batch. So if you have 40 emails to send out and this is set to 20, the first 20 go out as a group. What happens to the other 20? They go into a queue. The next time there is traffic on your forum (or the next time a system cron is triggered, if you are set up to use a system cron), phpBB will attempt to send out the next 20 emails in the queue. Set this to 0 and the queue goes away. There are some problems with this approach:
    • It’s slower because it takes time to send out an email to one user, wait for an acknowledgement from the mail server, then send the next
    • It’s possible that this will take so long that the process will time out, something more likely on shared hosting
    • If you hit some sort of outgoing email quota, not all emails will go out and you may not be aware there is a problem
  • Contact email address and return email address. These email addresses should exist as real mailboxes. Your web host control panel should have a feature allowing you to create email accounts. (You can also set up forwarders to send any email these accounts receive to one you will read.) In addition, it is highly advisable that these email addresses use your domain. Why? It lessens the likelihood that the email will be considered spam because:
    • It’s coming from an email address associated with the IP address of your domain
    • Because these email addresses actually exist it suggests the email is legitimate
  • Hide email addresses. If the same email is going to multiple recipients, the email addresses will be BCCed (Blind Carbon Copied) instead of placed on the TO field. Unless your host has a policy of not allowing BCCed emails to be sent, you should leave this set to Yes.

Verify your site name

ACP > General > Board configuration > Board settings. Verify that the Site name field is filled in. Some administrators will blank this out because they don’t want it to appear on their style, as they have substituted a logo for their site name. This is a mistake because outgoing emails, such as email notifications are not distinct. This causes them to look like spam because many emails use a same generic subject, a trigger for many spam emails. These emails will often be blocked from being sent at all, or placed in a spam folder if they are received by the recipient. If necessary, you can edit the overall_header.html file in your style’s template folder to hide or remove the site name. (Purge the cache afterward for the change to take effect.)

Verify your web host’s email policies

Since most phpBB forums are on shared servers, it’s important to know what limits there are if any for sending out emails. This can often be found buried in the host’s knowledge base, but sometimes it takes a support ticket to know exactly what policies apply. Based on this information you can tune phpBB’s emailing algorithm to match your web host’s policies. Policies often have constraints like:

  • Only X emails can be sent over Y minutes or hours and/or
  • A maximum of Z distinct email addresses can be sent over Y minutes or hours and/or
  • Email FROM addresses must be real email addresses associated with your domain and/or
  • Sending emails using BCC is/is not allowed

With virtual private servers or dedicated servers these rules may be relaxed or not exist. In any event, email spam is obviously a huge issue. It’s reasonable to assume that your web host will be examining outgoing email and blocking any email it thinks is spam. And if it blocks any emails from going out, it might block all emails from going out. Generally the only way to know for sure this is happening is to ask your web host, because you probably won’t be notified. Web hosts send out lots of emails, and they don’t want their email servers to be put on email blacklists. So their outgoing spam settings are usually set to high, assuming being extra paranoid is better.

Setting the correct email package size

Based on knowing your host’s email policies you can figure out a correct email package size. For example, if a maximum of 25 emails can be sent every 5 minutes, a good email package size would be 25, although it might be better to set it a bit lower, perhaps to 20. If 25 are allowed and you set it to 40, then it’s possible 25 will go out and 15 will not.

Using a system cron

If your email package size is greater than 0, then phpBB will create a queue of emails if necessary. So if you have 25 to go out but specify 20 in a package, 20 will go out and 5 will go into a queue. When do the next five go out? By default they will go out the next time there is web traffic to your forum. (ACP traffic doesn’t count.) So if your board doesn’t get much traffic, it could be hours or days later before the next 5 emails go out. This is not good if your users expect timely notifications.

A better way is to use a system cron. This helps send emails out on a timely basis plus it allows you to be congruent with your web host’s email policies. Creating a system cron is a bit tricky and may require someone with these skills to set it up. Of course you can hire me to do this for you. Procedures are outlined on phpBB’s Wiki.

To enable a system cron, go to ACP > General > Server configuration > Server settings. Set Run periodic tasks from system cron to Yes, then program the necessary cron job on your web server. This will disable phpBB’s default cron that depends on board traffic. Instead, depending on the interval you set with the cron job, a system cron built into your web server’s operating system will trigger periodic events that will process the email queue and other phpBB work.

In the above example, if we know the policy is not to permit more than 25 emails per five minutes, a cron job like this could be programmed to run every five minutes:

*/5 * * * * cd /path/to/board; php ./bin/phpbbcli.php cron:run

On shared hosting this often won’t work. See the Wiki for details, but something like this might work instead. (Note in this case you do not need to set Run periodic tasks from system cron to Yes. This way board traffic might cause email notifications to go out sooner, but if not the cron job will simulate a human hitting the board, which will do the job.)

*/5 * * * * curl -A=Mozilla/4.0 http://www.yourforum.com/forum/cron.php?cron_type=cron.task.cron_task

Troubleshooting outgoing email issues

If there are errors sending out emails, phpBB may trap these in its Error log: ACP > Maintenance > Forum logs > Error log

Sometimes though phpBB cannot detect problems. You may find clues in your web server error log, if one exists and you can read it. Sometimes you will see an error_log file in your phpBB root folder, or in the website’s root folder, or in a /log directory that can be found with the web host’s File Manager. Looking at it or downloading it for examination may show error messages that will point to the root of the problem.

In many cases though you can’t solve them. You will have to get your web host involved. They will have to look at email server logs to see if anything shows there and provide guidance.

Troubleshooting recipient email issues

If you are confident that emails are being sent to users, then most likely they are being received but not seen because they are ending up in a spam folder. In most cases there is not much you can do about this problem.

Most email programs allow users to create rules to automatically move certain emails in the spam folder into their inbox instead.

It’s possible that your domain has been blacklisted for sending spam, or what others think is spam. You can check to see if this is a problem. In many cases you can appeal to the blacklist site and get the block removed. blacklistalert.org is one such site.

Update on dealing with forum spam

Some time back I wrote about how to deal with spam forum registrations and spam posts. Since I wrote these posts, spammers have changed tactics and phpBB has a number of new solutions to help address these issues. So a brief update:

  • With the release of phpBB 3.2 Rhea, the phpBB group has integrated Google’s new reCAPTCHA. The old one had been thoroughly hacked. You may have seen this one already on other websites. It asks if you are a human and gives you a checkbox to click on. This is generally all you have to do to “solve” the CAPTCHA. So if you are running phpBB 3.2 you may want to use this Spambot countermeasure as it is simpler than the Question & Answer countermeasure, previously the best solution if it was done right. Since the old reCAPTCHA was eventually defeated by spammers, I suspect this new version will have limited shelf life too, so if you use it keep an eye on it and if it starts failing use something else.
  • Extension developer RMcGirr83 has released a Stop Forum Spam extension. It works on both phpBB 3.1 and 3.2. It works by querying the stopforumspam.com database. This should catch the vast majority of spammers, but it may let a few slip through. If you allow guest posting, it can also be configured to check guest posts.

The Cleantalk service remains an option. It costs $8/year for a website and requires the installation of an extension or modification (depending on your version of phpBB) as well as getting a key from the website to enable it. I have had one client with an issue with it falsely identifying a user as a spammer. I worked with them to address it. Otherwise, my clients have noted no issues and recommend it highly.

 

Removing and preventing spam posts

Note: this post was updated on February 10, 2019 to bring it up to date.

Note: this post was edited on February 3, 2018 to keep it up to date, due to its popularity.

Note: This post was edited on July 22, 2018 to discuss tools for removing spam posts.

Back in 2015 I promised a subsequent post on removing spam posts from phpBB forums. Before talking about how likely spam posts can be removed let’s first talk about how to prevent them in the first place.

Preventing spam posts

You may want to adopt one or more of these strategies:

  • In 2018, the phpBB group approved the release of the Akismet anti-spam extension. This service uses the popular Akismet service, which is essentially a huge database of IPs and domains that have been reported to have sent out spam. Akismet is primarily used for comments on WordPress blogs, but was tailored by the extension developer to also work with phpBB. While the Akismet service can be free to use, it is not necessarily free. It is free for personal sites and blogs. If your forum is on your personal, noncommercial website, then presumably you can use it for free, although you are encouraged to donate anyhow. The extension though is free to install and use, as is true of all phpBB extensions. When properly enabled, the service will check new registrations and posts and will disallow them if they meet the spam threshold. Note that as of this writing it has no tool to go through existing registrations and posts to find and remove spam.
  • Similar to Akismet is the Cleantalk service. It’s arguably more affordable than Akismet, at least if you don’t qualify for Akismet’s free tier. You pay $8USD a year to subscribe to the service. You will have to download the Cleantalk extension for phpBB. (At this time an extension for 3.1 and 3.2 is available. However I recommend getting the latest version from GitHub, as it has features that may not appear on phpbb.com for months.) Install it, then configure it to check all posts for spam before allowing the post to be posted. As a bonus, it can check for spam in the contact form if that is enabled. There is no CAPTCHA built into the contact form. This is probably the most effective solution currently available. Note: the newer versions of this plugin also have a neat feature called Spam Firewall that can be enabled. It basically keep spambots from hitting your forum in the first place, saving you bandwidth and server resources.
  • Do not allow guests to post. Fortunately, phpBB comes configured this way by default. If you actually want guests to post:
    1. ACP > Users and groups > Group forum permissions
    2. Select the forums that you want guests to post in and submit the forum.
    3. Select the role for guests for each forum. Probably you want Limited access but may prefer to be more expansive with guests and give Standard access. Then click Apply all permissions.
  • Use the phpBB stop forum spam extension. This will check the IP of the poster against a popular known spammer’s database, but only this one list. It’s not foolproof, but it’s probably a 95% solution. Note that this extension works only for guest posts, so a registered user’s IP won’t be checked to see if their post contains spam. One advantage over Akismet and Cleantalk is it never costs any money to use this database.
  • Use moderators. Find active and trusted users to help moderate your forums. You can make them global moderators or give them permissions to moderate specific forums only. Moderators also need to learn phpBB’s moderation procedures. In most cases it takes a human to truly identify a spam post.
  • Encourage users to report spam posts. You might want to create an announcement to draw people’s attention to this feature of phpBB. It’s easy not to see it. For every post in the top right corner of the post there is a small button with an exclamation point (!) on it. The user can identify the reason for reporting the post, which can be it is spam. This will flag it for moderators or the administrator to review.
  • By default newly registered users to have their first three posts go through the moderation process before they can post. If you do not have moderators set up, then you as the administrator will have to review and approve these posts. Follow phpBB’s moderation procedures.

Use better registration procedures

If your board is clean of spam, upping your spambot countermeasures can help ensure that no spambots register. A spambot that succeeds in registering can create spam posts.

  • With the release of phpBB 3.2, phpBB can be integrated with the second generation version of reCaptcha. You need to go to the reCaptcha site, select the version 2 reCaptcha with the checkbox Captcha and generate a set of private and public keys for your domain (if you don’t have them already). Then configure the plugin: ACP > General > Board configuration > Spambot countermeasures. Look under Available plugins for reCaptcha and press Configure. Once the keys are entered you have to enable the plugin.
  • If you are using phpBB 3.1, the best out-of-box solution is to use the Q&A countermeasure. Make sure the question is not easily retrieved with an “I feel lucky” Google search.

Removing spam posts

The latest version of the Cleantalk extension has tools that can help identify and remove spam users by checking the IP they use with their database. If it matches, you have the option to remove their accounts and with it all their posts. It is possible but unlikely that it will give some false positives, in which case using this approach may delete a lot of legitimate posts. It requires subscribing to their service, which costs $8/year as this is written. For more information, see this blog post.

Here are some other much more laborious means of identifying and removing spam posts:

An administrator or a forum moderator can manually remove any post he or she judges to be spam. Click on the little X icon in the top right corner of the post. If there is not much spam on your forum, this is generally the quickest approach.

If you allow guest to posts, a list of forums, topics and posts that have guest posts is useful. Administrators or moderators could then review these posts and delete them as needed. If you have phpMyAdmin, you can use it to run the following SQL to identify guest posts. (Select the forum’s database and then select the SQL tab.) Make sure you change phpbb_ as the table prefix if your config.php shows you have a different table prefix. The post text may look a little weird, as it is typically stored as HTML (phpBB 3.2) or in BBCode (previous versions), but it can be read.

SELECT f.forum_name, t.topic_title, p.post_subject, p.post_text
 FROM phpbb_forums f, phpbb_topics t, phpbb_posts p, phpbb_users u
 WHERE t.topic_id = p.topic_id and f.forum_id = t.forum_id AND p.poster_id = u.user_id and user_id = 1
 ORDER BY left_id ASC, t.topic_id DESC, post_id ASC

If older guest posts were valid but you notice a rash of spam guest posts after a certain time, you can see a list of posts on or after this time. In this example, January 1, 2016 is used.

SELECT f.forum_name, t.topic_title, p.post_subject, p.post_text
 FROM phpbb_forums f, phpbb_topics t, phpbb_posts p, phpbb_users u
 WHERE t.topic_id = p.topic_id and f.forum_id = t.forum_id AND p.poster_id = u.user_id and user_id = 1 AND p.post_time >= unix_timestamp('2016-01-01 00:00:00')
 ORDER BY left_id ASC, t.topic_id DESC, post_id ASC

The query will identify the forum, topic, post subject and post’s text. This query is ordered to present these posts in a way that is ordered the same way it usually is on the forum.

phpMyAdmin, which is generally available in your web host control panel, has an export capability. You could, for example, export this list as a comma separated values (CSV) value, import it into a spreadsheet like Excel and pass it out in a more human readable format to moderators for review. They will have to find these posts and delete them manually in phpBB.

Do not delete these using SQL, as you will mess up the topic post counts and possibly the number of topics in a forum count. Manually delete them on the view topic screen instead.

Fixing phpBB spam registration problems

(Note: since this post is frequently read, I updated it for phpBB 3.2 on June 28, 2017)

I am frequently sought out to help address issues with spam registrations on phpBB forums. The most typical spam problem I encounter is not spam posts, but spam registrations. The symptoms are lots of bogus usernames in your member list or inactive user list, often strings of random letters and numbers for the username, with a “website” listed in their profile that points to a spam site.

I am happy to fix these for you. You may wish to try some of the solutions below first. The solutions discussed apply principally to phpBB 3.0, 3.1 and 3.2.

Getting rid of existing spam registrations

Most of these “users” don’t bother to complete registration, simply want to leave a spam memberlist link and be gone. Consequently they show up as inactive users.

  • Inactive users can be deleted in the Administration Control Panel. ACP > Users and Groups > Inactive Users. Select the inactive users you believe to be spam registrations and in the little drop down box in the bottom right of the screen select Delete and press Submit. You may have to do this many times for many screens to get rid of them all. (Note the Mark All link at the bottom of the page. This can speed up things.) This is the safest approach.
  • You can also use the Prune Users function: ACP > Users and Groups > Prune users. A user that has not completed registration is not necessarily a spammer. Unfortunately, the Prune User function is not smart enough to be able to examine the profile website field to see if it contains data as this is often the key to filtering out likely spammers. However, if you prune a legitimate inactive user they can always come back to the board and register again. To remove these users it helps to look at your memberlist by date and see if you can figure out when the spam started. In the Prune users utility you can enter this date in the Joined field. Entering 0000-00-00 in the Last Active field essentially is the same as filtering inactive users only. You can select the option to delete posts for these users too but if they haven’t completed registration there should be no posts to delete.
  • You can globally delete these on the backend with SQL but beware: you may also delete legitimate new users that haven’t finished the registration process. Running any raw SQL statement is inherently risky, so backup the database first! Make sure you select the right table prefix (phpbb_ is shown in the examples) for the users table. Your config.php file contains your table prefix, which is usually “phpbb_” as well as the database you are using. If you have more than one application this will distinguish the one containing the data for your forum. You can use a tool (typically phpMyAdmin in your web host control panel) to issue SQL to delete these. Here’s an example of SQL for MySQL that will remove all inactive users with zero posts but with a website URL:
    • For phpBB 3.1 and 3.2:
delete from phpbb_users
 where user_id in
 (select user_id from (select u.user_id from phpbb_users u, phpbb_profile_fields_data p
 where u.user_id = p.user_id
 and user_posts = 0
 and user_type = 1
 and pf_phpbb_website <> '') as u)
    • For phpBB 3.0:
delete from phpbb_users
 where user_id in
 (select user_id from (select user_id from phpbb_users
 where user_posts = 0
 and user_type = 1
 and user_website <> '') as u)

Reducing spam registrations

Since most spam comes in the form of registrations and not spam posts, generally tightening up the registration process can reduce or eliminate these registrations. Many admins require administrator approval for each registration, but this becomes labor intensive. Most CAPTCHAs have been thoroughly hacked and are to be avoided. Here are some better alternatives:

  • Tighten up your spambot countermeasure to something more likely to work, like the question and answer one.
    • ACP > General > Board configuration > Spambot countermeasures
    • Under Available plug ins, select Q&A and press Configure
      • Create a question that is unique to the focus of your forum and won’t be guessed with a Google search. Create as many questions as you want. All must be successfully answered to complete registration.
      • Go back to Spambot countermeasures, select Q&A again and press Submit. This will change the countermeasure.
    • If this stops the spam, it’s probably safe to change it so the admin doesn’t have to approve every registration. An email verification is a good approach. ACP > Board configuration > User registration settings > Account activation > By user (email verification)
    • As a best practice or whenever you start to notice spam registrations again or quarterly change the registration question(s)
  • phpBB 3.2 supports Google’s new version of reCaptcha as a spambot countermeasure.
    • First go to the reCaptcha site and generate the public and private keys you will need for your domain.
    • ACP > General > Board configuration > Spambot countermeasures
    • Under Available plug ins, reCaptcha and press Configure
    • Enter the public and private keys from the reCaptcha site into the fields and press Submit.
    • Go back to Spambot countermeasures, select reCaptcha again and press Submit. This will change the countermeasure.

Some sites get hammered and even these steps are not enough. If this happens to you try these options:

  • phpBB 3.1 and 3.2: Install the Stop Forum Spam extension. This checks the IP and some other information of the user registering or the guest poster against popular blacklists, and if there is a match they cannot post. It’s not 100% perfect so some legitimate people may not get through and it’s also possible some spam will get through.
  • phpBB 3.1: Install and use the Sortable Captcha extension
  • phpBB 3.0: Install the Advanced Block Mod. Warning: this is a very complex mod to install and configure correctly. You will need to point it to more updated blacklists. You may want to have me install it professionally.
  • If you don’t mind paying $8USD a year, my customers report 100% success with Cleantalk. You will have to download the modification or extension, create an account on Cleantalk and enter your registration key in phpBB. It’s worth the time and expense.

I will cover how to remove spam posts in a future blog post.