Moving your forum to https

Should your forum transmit and receive data securely? Most forums don’t contain sensitive information, so you would think the answer would normally be “no”. A secure forum encrypts all communications between server and client. This would be done by changing the URL of your forum to use https (Secure HTTP) instead of http (insecure).

Once considered a nice-to-have feature, technology companies are nudging us content providers to use https. Google is primarily responsible for upping the ante. Back in 2014, Google announced that sites that send data securely would be ranked higher than those that did not, all things being equal. This is a pretty good incentive for site owners to respond, particularly if you are concerned about your site ranking. However, in 2014 moving to https was still a pain so lots of site owners decided to dodge the issue.

As with most things, going to https can be complicated and potentially expensive and/or time consuming. Fortunately, it’s less complicated than it was, and can even be free.

SSL vs. TLS encryption

To make https work, a digital certificate must be installed on your web server. Keys in the certificate are used to encrypt communications, by the server with a private key which is decrypted by the receiver with a public key provided when the connection is established. SSL (secure socket layer) or TLS (transport layer security) protocols are used to facilitate secure communications over HTTP. TLS is the newer technology and SSL is now seen less frequently because it is easier to hack. Whether using SSL or TLS though, it’s behind the scenes stuff. The user just sees https in the URL and assumes data going to and from your forum will be transmitted securely.

Shared certificates

Hosts often provide a shared certificate you can use. As the name implies, the certificate is shared with others, generally all domains on the same server as the one that you are on. While this works, it is ugly. First, hosts will issue “self signed” certificates. Browsers will not trust self signed certificates and will ask users if they want to trust the certificate. You generally pick an “advanced” link in the browser and give your browser permission to trust the certificate. This obviously will not inspire confidence in users coming to your site. New users may simply opt out of coming to your forum altogether, feeling it is untrustworthy.

Paid certificates

Web hosts will usually offer to sell you a certificate, generally for around $75/year. This is a convenient way to go if cost is not a concern. Some hosts will handle the logistics of integrating the certificate for you. Also, these certificates will be trusted by the browser, as they will come from a certificate authority the browser will recognize as trusted.

As you might expect there are various levels of certificates based on the level of trust you are willing to pay for. Higher class certificates require site owners to submit credentials to prove they own their domains and they are who they say they are. This is especially important in electronic commerce. Hence Amazon’s certificates will cost a lot more than any certificate you are likely to get. If you are doing electronic commerce on your site you might want to pay for a higher level certificate, which will require you providing credentials to the certificate authority. In most cases though forums simply need a low class certificate, enough so that the certificate is trusted by the browser by default.

Let’s Encrypt certificates

The hassle and cost of securing web traffic has become recognized as a general issue, leading to a project to make trusted certificates available for free. The Let’s Encrypt site will issue certificates for free that are recognized by all the major browsers. However, the certificates are only good for three months. Moreover, depending on your host, installing and renewing certificates can be a considerable hassle. For example, I use MediaTemple‘s Grid Service to host this site. It supports Let’s Encrypt, but it’s quite a pain to install and renew certificates. Other sites, like SiteGround, make it automatic. All things being equal, you might prefer a host that makes installing and renewing Let’s Encrypt certificates easy, especially if this is important to your site.

Configuring phpBB to use HTTPS

By default, phpBB assumes you will be using HTTP, not HTTPS. Once your certificate is installed and tested, it’s easy to change phpBB in the Administration Control Panel: ACP > General > Server configuration > Server settings. Then change server protocol from http:// to https://. What this does is change the links across the site.

Redirecting HTTP traffic to HTTPS

Even with a certificate installed it’s possible that you will get requests for forum traffic using HTTP. You may want to make all HTTP traffic use HTTPS traffic instead. You can see what type of web server you are using the Administration Control Panel: ACP > General > Quick access > PHP Information. Scan for “Server API”.

These instructions will work if your web server is Apache. Edit your .htaccess file in your forum (or to make it across the whole site, edit or create a .htaccess file in your web root) as follows. Place this code at or near the top of the file, changing mysite.com to your domain name:

RewriteEngine On 
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://www.mysite.com/$1 [R,L]

If you use nginx, use these instructions. If you use Microsoft’s IIS, use these.

 

 

Smartfeed 3.0.6 archive had unnecessary files in root folder

When I created the archive of Smartfeed 3.0.6, apparently some files that exist in folders were copied into the root folder. It didn’t hurt anything but it’s incorrect. I have republished the archive on this site. You might want to delete these files and reinstall it in /ext/phpbbservices/smartfeed.

Aside from this error there were no code changes.

It was always correct if you downloaded it from the GitHub Smartfeed page.

Removing and preventing spam posts

Note: this post was edited on February 3, 2018 to keep it up to date, due to its popularity.

Note: This post was edited on July 22, 2018 to discuss tools for removing spam posts.

Back in 2015 I promised a subsequent post on removing spam posts from phpBB forums. Before talking about how likely spam posts can be removed let’s first talk about how to prevent them in the first place.

Preventing spam posts

  • One the simplest current way to prevent spam is to pay $8USD a year and get the Cleantalk service. You will have to download the Cleantalk extension for phpBB. (At this time an extension for 3.1 and 3.2 is available. However I recommend getting the latest version from GitHub, as it has features that may not appear on phpbb.com for months.) Don’t pay for the service if there is no extension that works with your version of phpBB. Install it, then configure it to check all posts for spam before allowing the post to be posted. This is probably the most effective solution currently available. Note: the newer versions of this plugin also have a neat feature called Spam Firewall that can be enabled. It basically keep spambots from hitting your forum in the first place, saving you bandwidth and server resources.
  • Do not allow guests to post. Fortunately, phpBB comes configured this way by default. If you actually want guests to post:
    1. ACP > Users and groups > Group forum permissions
    2. Select the forums that you want guests to post in and submit the forum.
    3. Select the role for guests for each forum. Probably you want Limited access but may prefer to be more expansive with guests and give Standard access. Then click Apply all permissions.
    4. Then install the phpBB stop forum spam extension. This will check the IP of the poster against a number of popular known spammer lists. It’s not foolproof, but it’s probably a 95% solution. Note that this extension works only for guest posts, so a registered user’s IP won’t be checked to see if their post contains spam.
  • Use moderators. Find active and trusted users to help moderate your forums. You can make them global moderators or give them permissions to moderate specific forums only. Moderators also need to learn phpBB’s moderation procedures. In most cases it takes a human to truly identify a spam post.
  • Encourage users to report spam posts. You might want to create an announcement to draw people’s attention to this feature of phpBB. It’s easy not to see it. For every post in the top right corner of the post there is a small button with an exclamation point (!) on it. The user can identify the reason for reporting the post, which can be it is spam. This will flag it for moderators or the administrator to review.
  • By default newly registered users to have their first three posts go through the moderation process before they can post. If you do not have moderators set up, then you as the administrator will have to review and approve these posts. Follow phpBB’s moderation procedures.

Preventing spam posts through better registration procedures

If your board is clean of spam, upping your spambot countermeasures can help ensure that no spambots register. A spambot that succeeds in registering can create spam posts.

  • With the release of phpBB 3.2, phpBB can be integrated with the second generation version of reCaptcha. You need to go to the reCaptcha site, select the version 2 reCaptcha and generate a set of private and public keys for your domain (if you don’t have them already). Then configure the plugin: ACP > General > Board configuration > Spambot countermeasures. Look under Available plugins for reCaptcha and press Configure. Once the keys are entered you have to enable the plugin.
  • If you are using phpBB 3.1, the best out-of-box solution is to use the Q&A countermeasure. Make sure the question is not easily retrieved with an “I feel lucky” Google search.

Removing spam posts

The latest version of the Cleantalk extension has tools that can help identify and remove spam posts. It requires subscribing to their service, which costs $8/year as this is written. For more information, see this blog post.

Here are some other much more laborious means of identifying and removing spam posts:

An administrator or a forum moderator can manually remove any post he or she judges to be spam. Click on the little X icon in the top right corner of the post. If there is not much spam on your forum, this is generally the quickest approach.

If you allow guest to posts, a list of forums, topics and posts that have guest posts is useful. Administrators or moderators could then review these posts and delete them as needed. If you have phpMyAdmin, you can use it to run the following SQL to identify guest posts. (Select the forum’s database and then select the SQL tab.) Make sure you change phpbb_ as the table prefix if your config.php shows you have a different table prefix. The post text may look a little weird, as it is typically stored as HTML (phpBB 3.2) or in BBCode (previous versions), but it can be read.

SELECT f.forum_name, t.topic_title, p.post_subject, p.post_text
 FROM phpbb_forums f, phpbb_topics t, phpbb_posts p, phpbb_users u
 WHERE t.topic_id = p.topic_id and f.forum_id = t.forum_id AND p.poster_id = u.user_id and user_id = 1
 ORDER BY left_id ASC, t.topic_id DESC, post_id ASC

If older guest posts were valid but you notice a rash of spam guest posts after a certain time, you can see a list of posts on or after this time. In this example, January 1, 2016 is used.

SELECT f.forum_name, t.topic_title, p.post_subject, p.post_text
 FROM phpbb_forums f, phpbb_topics t, phpbb_posts p, phpbb_users u
 WHERE t.topic_id = p.topic_id and f.forum_id = t.forum_id AND p.poster_id = u.user_id and user_id = 1 AND p.post_time >= unix_timestamp('2016-01-01 00:00:00')
 ORDER BY left_id ASC, t.topic_id DESC, post_id ASC

The query will identify the forum, topic, post subject and post’s text. This query is ordered to present these posts in a way that is ordered the same way it usually is on the forum.

phpMyAdmin has an export capability. You could, for example, export this list as a comma separated values (CSV) value, import it into a spreadsheet like Excel and pass it out in a more human readable format to moderators for review. They will have to find these posts and delete them manually in phpBB.

Do not delete these using SQL, as you will mess up the topic post counts and possibly the number of topics in a forum count. Manually delete them on the view topic screen instead.

phpBB 3.2 Rhea, third look

(Read part 1 and part 2 if you haven’t.)

Updating from 3.0 or 3.1

One thing that is unclear about phpBB 3.2 is how to update to it. The phpBB group itself may be contributing to the confusion. On the launch page they say:

With our brand new installer updating will be easier than ever in phpBB 3.2! Upload a single folder to your board and all your files will automatically be replaced.

This implies that it’s already there. But if you are upgrading from 3.1 no interface exists. Rather, you follow procedures similar to updating from 3.0 to 3.1, found here. The easier way of updating will apply to subsequent releases of phpBB 3.2. There is no 3.2.1 released yet, so you can use the new improved updater yet.

In addition, you used to run /install/database_update.php to update the database. Now you run /install/app.php/update instead and select Update database only. You can also run /install and select the update tab and do it that way.

Other things to note:

  • Quotes are improved. To quote someone you generally click the “Reply with quote” button on the post you want to quote. Doing this creates an enhanced BBCode quote tag with additional attributes that include the post_id, the post time and the user_id. When you submit your post, the quote now indicates the this information in the quote and via an up arrow link can take you to the referencing post.
  • The notifications system has been rewritten to be faster. Since notifications are created when posts are made, you should notice much less delay time between submitting the post and the topic refreshing.

Smartfeed 3.0.6 released

The extension has been submitted for phpBB extension team review. You can download it here.

Of note:

  • All corrections required by the extensions review team have been addressed. All code was reviewed in PhpStorm to fix problematic issues and remove unneeded variables.
  • Supports phpBB 3.1.9 through 3.2 (Rhea)
  • Fixed bug that did not provide the encryption key (e parameter) with the embedded IP in the Smartfeed URL if Require IP Authentication is turned on in the ACP, leading to an erroneous error message if the URL for the feed is used.
  • Ability to use the ACP interface now requires the acl_a_extensions permission instead of acl_a_board permission
  • Copyrights changed to 2017
  • Containers are used to fetch global variables
  • jQuery UI library added to support enhanced dialog boxes
  • Where appropriate Javascript events now written in jQuery
  • CDATA removed from templates, as they are not needed in HTML5
  • Javascript without template variables moved into .js files
  • Host URL updated to use www prefix
  • Some previously allowed tags in Safe HTML were removed as they are deprecated

phpBB 3.2 Rhea, second look

With phpBB 3.2.0 Rhea now officially released, I have some additional observations about this minor release of phpBB beyond my last post.

  • All libraries needed are part of the archive, so there is no reason to run Composer before installing. This is good and makes life easier but there are more files to upload and the archive is getting fatter: 7.5MB zipped, 33MB unzipped.
  • The installer’s progress bar is slow to give feedback, leading perhaps to the impression that it has stalled. Wait and your patience will be rewarded.
  • Upon successful installation there are two options/extensions enabled. The first allows you to send your statistics to phpbb.com to improve the product. This was there previously but now it’s a simple checkbox and it can be enabled or disabled later. It has its own item in the Administration Control Panel: ACP > General > Server Configuration > Help support phpBB. More unusual is a new phpBB Group developed extension that integrates with VigLink. If you enable this extension, it adds URL tracking information when you or a forum user shares a URL. It allows the phpBB Group to generate some small revenue when people click on these links and then click on a targeted ad. You can uncheck this but it is enabled by default. It too has an item in the ACP: ACP > General > Board configuration > VigLink settings.
  • When installing a new board, you get a default category, forum and post, but no search index is populated with the post. Instead, you are reminded to create a search index. This allows you to create a different search index type if you want, but it’s an extra and new step to the installation process.
  • If you edit a post, the edit window is smaller, in that it shows fewer lines. You can drag the window to make it bigger but some people will find this change a bit irritating.

I had done a smoke test with my Smartfeed and Digests extensions. I posted about Smartfeed here, and Digests here. Look for new releases of both in the coming weeks and months that will support phpBB 3.2.

With the release the phpBB Group has published a features page with the major new features.

I’ll post more observations as I use it more.

 

phpBB 3.2 Rhea, first look

I’ve been waiting for the dust to settle to study phpBB 3.2 (Rhea). It is scheduled for release on January 7, 2017. So I finally installed a prerelease version with presumably almost all the bugs fixed. Here’s my first look:

New features

There’s not much new or sexy about phpBB 3.2 compared with phpBB 3.1, but it depends on what you are looking for. New features include:

  • Support for emoji in posts. You can cut and paste or simply type your own emoji shortcuts into posts and the emoji will render. You can find a comprehensive list of emoji shortcuts here. For example, in a post you can enter :grinning: and a scalable grinning emoji should be rendered.
  • Supports PHP 7.1. PHP 7 is a quantum leap in speed for the PHP script processor. Most sites can expect a 100% improvement in how quickly PHP will parse and render code written in PHP.
  • Global announcements are no longer an administrator only privilege.
  • FontAwesome support. FontAwesome allows scalable vector fonts and icons, controlled by cascading stylesheets. For example, if you have a FontAwesome icon of an airplane, the icon will scale to size as you increase magnification on the page without losing detail. In addition, FontAwesome allows the size, color and shadow of the font to be changed on the fly using CSS … no jQuery magic required anymore.
  • New installer. This is backend stuff. Installing phpBB looks a bit different, and looks spiffier. Before I could it install, however, I first had to run PHP from the command line to kick off a run of PHP’s composer software. Composer is used to fetch the third party libraries that phpBB uses, presumably to get a current version of these libraries. Previously they were bundled into the phpBB archive you downloaded. It’s unclear to me if this is something you will have to do when phpBB 3.2 is released before it is installed. If so it will prove an obstacle to many casual forum administrators, since they may not be familiar with working from a command prompt and it may not be an option on shared hosting. The installer’s command line interface has also been reworked. I have not yet investigated what’s new here.

Other changes of note

  • The default prosilver style looks a little bit darker, and the icons have been reworked and look a bit different, and are seamlessly scalable because they will use FontAwesome.
  • No subsilver2 support. Someone developed a subsilver2 style for phpBB 3.1 but it was not responsive (scalable for mobile devices). With 3.2 only responsive styles are supported. subsilver2 uses HTML tables to layout content, which is not responsive, hence it is not supported.
  • The reCAPTCHA spambot countermeasure has been updated to use Google’s latest (presumably the checkbox where you assert you are a human). The old one had been hacked, so this is encouraging. Perhaps it will be useful as a spambot countermeasure again.
  • New events. This is only of interest to extension authors. They have more places in the code and in templates to hook in additional functionality.
  • BBCode overhaul. TextFormatter has been integrated into phpBB to render BBCode, making a lot of longstanding BBCode related bugs go away.

Some cautions

  • You should first upgrade phpBB from 3.1 to 3.2 before you change your web host control panel to use PHP 7. (Note: if you have other PHP applications installed, make sure they can handle PHP 7!)
  • You must run at least PHP 5.4 if you want to run phpBB 3.2, so this may require a web host control panel change. Make this change before upgrading phpBB.
  • While most 3.1 extensions will probably work fine in the 3.2 architecture, some will require changes if only to assert that they will work under 3.2. I have not tested my Digests and Smartfeed extensions with 3.2 yet, but I expect no issues. I will have to issue new versions since ext.php will have to allow phpBB 3.2 to be used.

phpBB 3.1 end of life support

  • The support forums on phpbb.com will provide support for phpBB 3.1 through the end of 2017.
  • New releases of phpBB 3.1 are expected as needed through July 2017. A phpBB 3.1.11 release is in the works.

Should you upgrade now?

In general it’s a bit dangerous to be first out of the gate when phpbb.com releases a new minor version of phpBB. Unless there is a compelling reason otherwise, I’d wait a few months before upgrading to 3.2 but if you use the prosilver style with no extensions it might be worth installing when available. Check your styles and extensions and make sure each supports 3.2 before upgrading, or be prepared to use a standard style and have incompatible extensions disabled.

If you would like me to upgrade you to 3.2 contact me. Most upgrades cost $30USD.

December 2016 work summary

December was a fairly slow month for incoming work. Likely the holidays had something to do with it. All client information has been anonymized. Among the work I did in December were these jobs:

  • I upgraded forum from phpBB 3.1.9 to 3.1.10. I also updated the forum’s English, Danish and German language packs. There were no phpBB 3.1.10 language pack updates for Swedish or Norwegian, which were also available as language options.
  • My digests mod (version 2.2.26) installed on a phpBB 3.0.14 forum, but digests did not appear to be going out. To troubleshoot, I sent output from the cron to my email address and it reported that digests were going out. Digest logging was turned off so there was no evidence that digests were going out. I turned on logging. I then upgraded digests to version 2.2.27. The client chose not to upgrade to 3.1 because the Boardwatch mod had not been converted to an extension.
  • I upgraded forum from phpBB 3.0.12 to 3.1.10. The default prosilver style was used. However, there were issues during upgrade. I had to add database columns to various profile tables and give some columns a default value so the database_update.php program could complete. I manually removed a number of dead module links, some via the database. I reapplied the logo. Then there were issues with too many redirects. Changing the redirect in cPanel to point to index.php worked and was acceptable to the client. Cleantalk extension installed and enabled.
  • No users were able to login to a forum. My investigation found that the forum had malware on it, and all .php files were affected. There were PHP eval() statements at the top of all the .php files. I spent much time working with the web host simply to get FTP access. Then I overwrote files with a 3.0.8 reference to remove the malware. However, lots of language packs were installed that were still infected with malware. I deleted all of the language packs except for British English, changed the database so all users and the board used British English and I was then able to login, with some errors due to PHP 5.4 on host. I was then able to upgrade forum to 3.1.10. I installed the prosilver instead of the previously used subsilver2. I reapplied the logo. File backup archives were left in forum root folder.
  • Client wanted to upgrade their forum but could not because Joomla was being used to serve most of the site’s content. Client is using an old version of Joomla but wants to move it to WordPress because a Joomla upgrade was problematic and the current version of Joomla would not work if PHP was upgraded. The existing phpBB forum is version 2.0.21 and the client wants to upgrade to the latest version: 3.1.10. Can’t upgrade to phpBB 3.1.10 as it requires PHP 5.3.3 or higher, so Joomla must move into WordPress first. I installed WordPress in a /wordpress directory. I found a plugin that will convert Joomla posts and images to Wordpress. I installed the WordPress Slovenian language pack. Client has friend that will take care of styling in WordPress. Tried various times to import the Joomla posts into WordPress with option to import media with duplicate names. It worked eventually with some back and forth with the WordPress plugin developer. This work is still ongoing.
  • I upgraded forum from phpBB 3.0.12 to 3.1.10 using the default prosilver style, no logo. I was careful not to delete some special images folders.
  • Finishing a project that went through a proof of concept phase in September. Work was to move the production forum to new host in the process moving from Windows hosting to Linux hosting. I moved folders for files, images, and store from phpBB 3.0 Windows production host, all other 3.1 files can from a test instance that was set up. I uploaded install folder to upgrade the database. I then ran database_update.php. However, there were issues. I had to add columns to two profile tables, create a login attempts table, and code around a migrator program so prune users would work successfully. I wasn’t able to login to new forum due to the proprietary CMS to check things out. Default prosilver style applied. The next day client redid the upgrade and it did not have the upgrade issues I had. Work seems to be done.
  • A client encountered a baffling white screen attempting to update forum to 3.1 on Windows hosting. Initial problem was I had to change database to use default prosilver style to make an error message go away. Then I discovered the cache folder was not writeable. Could not change file permissions in web host control panel. Error when trying to do so. The underlying problem is that Windows file permissions work differently than Unix file permissions, and the web host control panel would not allow them to be set the way phpBB requires them to be. I suggested moving to Linux hosting or I could file some support tickets. Likely an older version of PHP was being used on the Windows hosting, and that was causing the upgrade white screen since the older version does not support PHP namespaces, which would trigger an error during the upgrade. Client ordered Siteground hosting at my recommendation. I rehosted the forum, website and database once Siteground login issues were figured out. I then upgraded the forum from phpbb 3.0.12 to 3.1.10. I renamed index.php in web root folder so static page was served. Reapplied logo. Everything is predictable and faster and the client is glad to be off Windows hosting.