Fixing insecure content issues in phpBB

Updated December 27, 2018 to correct some things based on new information.

So you’ve decided to use HTTPS for your forum to show your content securely. This is good and it’s not too hard a thing to do in most cases. Everything looks good but sometimes you notice on browsers like Chrome the little green lock icon up on the URL field disappears. What’s going on? If you investigate by clicking on the icon you can usually figure out what’s going on: there is some insecure content on the web page.

What is insecure content?

Insecure content is content embedded on a web page that is delivered insecurely, i.e. from a web server using http instead of https. Usually these come from external sources, and are typically externally hosted images that are served insecurely.

One way to investigate these is to view the HTML source of the web page. Use the Find feature to scan for URLs with http:// instead of https://. The issue occurs with embedded images like this:

<img src="http://www.externalwebsite.com/myavatar.jpg />

If all these URLs could be changed to something like:

<img src="https://www.externalwebsite.com/myavatar.jpg />

then all would be well, that is if the external website supports https.

How do you fix these problems? There are typically two places where these problems manifest:

  • In post text
  • In the user’s avatar

Here are some approaches you can use to solve to fix the problem:

Use the Image Redirect extension

As of this writing the Image Redirect extension is a Beta release, so it is not recommended that you install it on a production system. This extension also requires that you set up a proxy server on your web server, not a trivial tasks and something you may not be able to do on your class of hosting. Camo Proxy is one example of a proxy server you can install. What this extension does is scan the page for these external image URLs, fetches them using a proxy and changes the URL so that it is served from your proxy copy, which will be on your machine and served securely. In theory this extension should solve all issues like this. Note that it takes some time to create a proxy image if it is not cached and this adds some small overhead, which may slow page rendering.

Fix the embedded URLs in your database

This works by changing the URLs in your database. You scan for http:// and replace it with https://. Using this approach has some limitations:

  • The server serving the remote content may not have https installed. What generally happens is the image is not served and a white box appears instead. This could make lots of posts look off or unacceptable, particularly if these images are large.
  • While it corrects existing URLs, it doesn’t prevent someone from doing the same thing in the future.

If you can live with these limitations, you can fix it in the database. This approach assumes you have MySQL or MariaDB as your database and that the REPLACE function is available. It also assumes you have phpMyAdmin or a similar way to issue SQL (Structure Query Language) commands to the database. In phpMyAdmin, there is a SQL tab where you can type in and execute SQL. Just make sure you use a SQL tab for your database.

There are two tables that typically need fixing: phpbb_posts and phpbb_users. Steps:

  1. Disable the forum
  2. Backup the forum’s tables. Make sure it is a complete backup by downloading the extract, uncompressing if if necessary and looking at the end of the file. There should be SQL in there populating the phpbb_zebra table at the bottom of the file.
  3. Use phpMyAdmin or a similar tool to go into your database. If you are not sure which database you need to modify, look at your forum’s config.php file. The database name is in the file.
  4. You can examine the extent of the problem by first looking at each table. In these examples I assume your table prefix is phpbb_. The config.php file contains the actual table prefix, which may be different.
SELECT post_text FROM phpbb_posts WHERE post_text like '%IMG src="http://%';
SELECT user_avatar FROM phpbb_users WHERE user_avatar like '%http://%'
  1. To actually fix these, use the following SQL:
UPDATE phpbb_posts set post_text = replace(post_text, 'http://','https://') WHERE post_text like '%IMG src="http://%';
UPDATE phpbb_users set user_avatar = replace(user_avatar, 'http://', 'https://') WHERE user_avatar like '%http://%'
  1. Reenable the board
  2. You might need to purge the cache, but it should not be necessary.

After these steps, some users may notice that their avatar no longer serves and there is a big, ugly white space instead. They may try to change the URL in their Avatar settings back to http:// to restore it, in which case the problem may recur. This option can be disabled (see below). In general they should be encouraged to upload an avatar so it can be served from your web server, which will then serve it securely.

Preventing future insecure content

For avatars, the issue is due to allowing remote avatars. This can be changed: ACP > Board configuration > Avatar settings > Enable remote avatars > No

For posts, you can remove the permission to use the [img] BBCode. The easiest way to do this:

  1. ACP > Permissions > Group forum permissions > Registered users group > All forums
  2. For each forum, click on the Advanced permissions link, then the Content tab.
  3. Set Can use [img] BBCode tag permission to Never. Note: this will affect everyone, including special groups and administrators. If you want to have it affect only registered users, set it to No instead. Other groups however may retain the permission to post embedded images. You may want to use this pattern on other groups you have defined. 

Fixing blank spaces where embedded images should appear

Since blank space represent placeholders for external images that no longer exist, the URL may need to be corrected. You can try the MySQL Replace function above if you know the new pattern to use.

Alternatively, you can install the External Images as Links extension. This will substitute a clickable URL for the image. It’s likely the URL will lead to HTTP 404 error (not found), but it at least resolves the blank space image in the post.

 

October 2018 work summary

I completed a major project this month, and that represented the bulk of my work. This nonprofit organization has been my clients for nine years now. Earlier in the year I convinced them they needed to move their forum off of phpBB 3.0 to the latest version. As they are surgeons it was hard to pin them down so it took months of sporadic back and forth. It’s nice to finally bring this home and collect a nice four figure paycheck for all the work. Their old forum had a number of modifications and code that I had inserted into the base code to do things like send an email notification with topic and post text to all members whenever a post was made. That won’t work with phpBB 3.2. It took a lot of experimentation to figure out something that was acceptable. Some of the things I did for them:

  • Used DavidIQ’s mailing list extension to handle the post and topic notifications with the post text. I had to tailor it to make it send emails to all registered users, rather than just one email address. I also had to change the template text to meet their requirements.
  • Installed five different styles and customized each to meet various styling requirements. I used the Quick style extension to give users an easy way to change styles on the navigation bar.
  • Figured out a way that users could upload and embed “web ready” videos without needing to use the Quicktime plugin. IMHO this is a serious deficiency of phpBB that it won’t use the <video> tag for these formats like .mp4 and .ogg. Basically this meant hacking the attachment_body.html template. I also had to create a MP4 attachment type, add it to the Flash group, enable the Flash group and take other formats out of the Flash group. I also had to enable the documents attachment group. This allows other video types to be uploaded, however they have to be downloaded to be actually played. The general idea is to encourage these surgeons to upload MP4 files only.
  • We quickly discovered that when MP4 videos were uploaded they were usually uncompressed. A 12 second video was 19.1MB! I found ffmeg, a command line utility that allows videos to be compressed, resized and moved from one format to another. I integrated ffmpeg into their virtual server and wrote a PHP script that once an hour looks for new videos. If it finds an uncompressed MP4, it compresses it using the “fast” method and updates the metadata for the file stored in the database. I used ffprobe to examine the video’s metadata, asked for it to output the data in a XML format, then used PHP’s DOM object to grab the information I needed. It took more than a day to write it, but it will save them lots of storage space as well as make videos faster to load and play. Given these videos are of surgery, they needed to be kept private so hosting on sites like YouTube was out of the question. The whole forum is tightly locked down to members only, and uses HTTPS to encrypt all communications.

There were lots of extensions installed as well in addition to other minor requirements addressed. Some extensions provided functionality identical to the mods they had, such as the Recent Topics extension. In a few cases they chose to drop the functionality of their old modification.

In any event, this project is largely closed although I expect a few bugs will manifest themselves in production use I will have to fix.

Other work in October:

  • I updated a phpBB from from version 3.2.2 to 3.2.3. Installed a beta Medals extension.
  • Spent two hours of labor on digest emailing issues. Since the hosting was GoDaddy, I pretty much knew what the problem was: GoDaddy blocks outgoing emails it thinks might be spam, and doesn’t bother to tell the client they are doing this. It basically takes a technical conversation with them and to get the client to certify that they really don’t have any inappropriate content or malware on their domains.
  • Spent time troubleshooting an issue on why no posts were being created on a forum. The basic issue was low board traffic. Also updated phpBB from version 3.2.2 to 3.2.3.
  • Answered some questions on user statistics and most users online. On the client’s old version of the forum, the number of users included a hack that added inactive users and bots. They really wanted that back so I changed /includes/acp/acp_main.php to include all types of users, and warned them this adds a complication with updates and upgrades.
  • Updated two domains for the same client from phpBB 3.2.2 to 3.2.3.
  • Client needed to change the domain registrar from GoDaddy to another registrar for two domains. Helped her with this obscure process.
  • Upgraded a vanilla prosilver style forum from phpBB 3.1.6 to 3.2.3. The upgrade was very slow and seemed to hang. I tried a few hours later and was able to get it to resume and finish. There was an issue related to moving to a new server and having PHP 7 installed, which triggered certain errors on phpBB 3.1. Replaced logo.
  • Upgraded another vanilla prosilver forum from phpBB 3.1.10 to 3.2.3. I changed the spambot countermeasure to reCaptcha V2. I added latest Cleantalk extension. I changed user registration settings to user gets email. Changed PHP to version 7.