Blog

January 2020 work summary

I did plenty of work in January, just not so much for clients. That business was slow, but picked up toward the end of the month, so much so that at the moment it looks like my plate in February should be full.

Why was I busy? I was writing and editing a book on phpBB administration. The introduction of phpBB 3.3 Proteus on January 6th added work, as I needed to update the book to talk about phpBB 3.3. As I noted, its feature set makes it more of a micro release than a minor release, so that part was good. But by changing the phpBB logo and a few other things, I had to recapture fifty or more screenshots, which was tedious and time consuming. Then of course the text had to be revised in places which also meant reading it through twice more. Right now it stands at 305 pages on 8.5 x 11 inch paper.

I reached a milestone of sorts when I turned it over for editing on Thursday. I should mention that I am using Maria Williams, so if you need an editor, hit her up! It will take her a couple of weeks for her to do her magic, then I have to figure out how to publish and market the book. I need cover art. Publishing will likely be on demand, and I expect most people will want electronic copies. I also need to figure out how to price it. It’s a sizable capital investment of time and money, so it needs to be profitable. I expect to offer some sort of discount for the first ninety days and for my existing clients. The last book published on the topic was written in 2006, so it described phpBB 2, so the market is out there. Also, that was a user guide, not an administrator’s guide.

Otherwise:

  • Added two users and forced a reactivation of two other users for a commercial client, who outsources this work to me.
  • Updated a board from phpBB 3.2.7 to 3.2.9. Also, I created a new forum with a forum logo on the board. I resized and placed a forum image logo with the forum, and put it in the requested place on the index. I gave the forum permissions similar to another forum. I investigated issues in the error log. Most of these errors seems to be due to debug being turned on.
  • More analysis for a client I did work for in December. I generated a list of users and email addresses on his archived forum that had contributed at least 1 post but whose usernames were not on the current forum. He will use the list to invite them to come to the new board. I installed a release candidate extension to creating lots of user accounts in bulk, but it didn’t work as advertised. I did some styling changes. I upped base font 1 pixel to 12pixels. Client wanted to know if my digests extension can show text only. Yes, and I sent him a sample. He wants digests generated by my extension to look similar to someone else’s digest. I deleted a user defined moderator group. I provided guidance on using the ACP Add User extension so lots of old users could be added. He would like to delegate the work to someone else, so discussed how this could be done with ACP permissions so their access was tightly limited in the ACP.
  • Client’s board wouldn’t come up on Bluehost hosting. After some analysis, I determined he had a phpBB 2.0.19 board and that only PHP 7 was available. He did have attachments. I converted the board to phpBB 3.3.0. Issues logging on were solved by removing the domain’s cookies, clearing browser cache and resetting the password, which was set back. I moved the old board into forum_old folder. I created the search index, disabled contact page, added Invisible reCAPTCHA. The resulting board is very basic. It may need some customization, styles and extensions installed. Later there was additional work. The client needed some gallery software. The phpBB Gallery extension is not being maintained and is not reliable, so looked for a solution outside of phpBB. I tried one solution, then tried another. Piwigo seemed to work after I installed a Community plugin/extension. This allowed users to upload their own pictures.
  • I updated a board from phpBB 3.2.0 to 3.2.9 on generic prosilver style. I updated my digests extension to version 3.2.17 and the Cleantalk extension to version 5.7.2.
  • I upgrade board from phpBB 3.2.7 to 3.3.0. I updated the prosilver SE style to 3.2.9 and put the old logo back. I changed the spambot countermeasure to reCaptcha V2 invisible, since spam registrations were a complaint.
  • Client wanted active topics in red on the navigation bar. First tried installing the Advanced Active Topics extension (release candidate) but couldn’t get it to show. I then installed the Recent Topics extension as a trial, but client didn’t like it, so I created a special link on the navigation bar, highlighted in red, moving it from the Quick Links menu.
  • I upgraded a board from phpBB 3.2.8 to 3.3.0. I replaced the logo.I checked all the extensions and they seemed to work on phpBB 3.3 with no issues, but there was no test case for change post time extension to verify it worked properly.
  • More work for a client on spam. Used Prune Users feature to remove users and their posts who registered after Feb 10, 2018 when spam seemed to start in earnest. Reenabled board and suggested ways to mitigate the problem in the future with moderation, digests, etc. This is discussed in more detail in this blog post.
  • This work is still ongoing. On Hostgator hosting, I created space for five domains and set up DNS for these domains. I set up five boards on five subdomains of these domains. All have a basic configuration, registration disabled, reCaptcha enabled, prosilver_se style applied, contact form disabled, and forwarder set up to redirect to client’s email address. Set up https redirection by installed WordPress SSL plugin on the main blog. Next, I cloned four WordPress instances based on work on one domain, whose theme and plugins client wanted to replicate. I did this by copying the files and database, changing the wp-config.php file to point to the new database, and tweaking the site information in the wp_options table. This saved a lot of time and work for the client.

 

Getting rid of old spam is hard

phpBB is now pretty good at keeping spam users from registering and posting. In the phpBB 3.0 and 3.1 days, its defenses turned out to be pretty weak. The GD Image spambot countermeasure (still the default) was easily hacked. phpBB has at least added settings to let you tune it better, making it harder to hack. It also started supporting Google’s reCAPTCHA, but the version in phpBB 3.1 was quickly hacked and phpBB was not agile enough to quickly integrate its versions 2 and 3 reCAPTCHAs.

This led to the an inundation of spam on certain forums, mostly bogus spam registrations but also lots of spam posts in some forums. Some administrators countered by requiring all new users to be approved by an administrator. But when inundated with hundreds of these in a short period of time, it’s a hassle to delete them all, or discern the real new users from the spam ones. For a few years, I made quite a bit of money removing spam for clients.

With phpBB 3.2 things slowly got better, at least if administrators used best practices. Best practices were to use reCAPTCHA version 2 “I am not a robot”, or the Question & Answer, providing the questions were sufficiently difficult. A malicious human could still take the time to solve the questions, but these were unusual. There were also a few extensions that could help. The Sortables CAPTCHA was one of the more useful ones.

My go to for years has been the Cleantalk extension, which requires subscribing to their service. But now there is also an Akismet spam extension, which also requires a subscription, which can be free for personal sites.

All this is good at preventing spam, but how do you get rid of months or years of spam posts? That was my dilemma this week working with a client.

The latest version of the Cleantalk extension has a feature that removes spam users and their posts. But I discovered it has a few serious limitations:

  • It bases its judgment based on the IP of the poster. The user’s last IP is stored automatically. It doesn’t examine the post text. Over time, IPs that used to be marked as spam get cleaned up, and when this happens these IPs are no longer flagged, so spam registrations aren’t caught.
  • Its interface for finding these users is slow and can easily time out, which means sometimes it can’t succeed. It also lacks pagination.

Why this particular client ignored this problem for a few years, I don’t know. But cleaning up the database was a big challenge. The only real way to do it is to manually look at every post and flag those that were spam, then use moderator tools to get rid of them.

This was time prohibitive, but if these could be removed presumably people would start posting again and Google would rank the site as legitimate again, bringing in new people.

The next best solution I found was to try to identify when the spam started. This took quite a bit of analysis, but looking in the most posted forum on the board it looked like it started on Feburary 10, 2018. So I used phpBB’s Prune User feature to remove users and their posts that registered after the spam started.

This seems to have gotten rid of the spam. But it also removed accounts of some legitimate users, and their posts as well. Those who had accounts before then were unaffected and their posts remained.

phpBB needs a real solution which so far doesn’t exist.

But I think I have found a solution … if I write the extension. If you are an extension developer, please go ahead and develop it, just tell me so I don’t waste my time.

It turns out that the Akismet, the biggest solution out there and used widely in WordPress to moderate comments, has a Submit Spam API. So in theory, if you pass the needed information to it including the poster’s IP and the post text, it can render a judgment on whether it is spam or not. If these posts can be flagged, they can then be removed.

One possible issue is that the service requires sending it a User Agent string. phpBB does not store this. Perhaps a fake user agent string could be supplied, but would this render a correct judgment? If no, this solution wouldn’t work. Also, it requires an Akismet key to use, which might require some boards to purchase the key. This may be a limiting factor for some.

As I have time I hope to see if this is a viable approach finding and removing spam posts in phpBB.

Selective mass emails version 1.0.9 (Release Candidate) released

Only minor changes were made, but it is tested on phpBB 3.3 and works on that and phpBB 3.2. Details here.

It can be downloaded from the GitHub branch or from the extension’s page. If downloaded from GitHub, make sure to disable the extension first, remove the old filed and place the files in the /ext/phpbbservices/selectivemassemails directory.

 

phpBB 3.3 is a pretty minor minor version

So I’m starting to port my extensions to phpBB 3.3.

I’m surprised by how easy it has been, at least compared with porting my extensions from phpBB 3.1 to 3.2. This is because not much has changed between 3.2 and 3.3.

In moving from 3.1 to 3.2, there were all sorts of changes under the hood that made porting extensions challenging. The way posts were encoded in the phpbb_posts table was changed, which was a big deal. FontAwesome support, which was very limited, was introduced, meaning we could use these in our templates instead of embedding images. A new TWIG library meant that the old way we extension authors inserted dynamic data into pages changed pretty fundamentally. We were asked to use a new template syntax in our templates. Template library calls also changed quite a bit. In short, 3.1 to 3.2 required quite a bit of reengineering, both of the phpBB base code but also of its extensions.

For 3.3, there’s not much upgrading of the database needed. Only one table is altered: phpbb_users, to change the properties of the reset token. The content in the phpbb_extension_groups table is changed. The Flash media group finally goes away (Flash is obsolete) and any non-Flash files in the group are now assigned to the downloadable files group.

As noted, you can use more Emoji codes in 3.3 and can put Emoji in a topic title. But this doesn’t alter the database. I have to test for these in my extensions, but so far it’s not a problem because the existing function I used to display these fields handles it for me. So nothing to do there.

So phpBB 3.3 feels more like an update to phpBB 3.2 than it does a new minor version. On the whole, the new functionality is quite minor. The main thing I have to do is test with PHP 7.4, since phpBB 3.3 now supports PHP 7.1.3 to 7.4 only. Getting my extensions to a stable 3.2 place before upgrading PHP is turning out to be much more work than retrofitting my extensions.

So yea for that.

Smartfeed 3.0.13 released — supports both phpBB 3.2 and 3.3

You can read the release announcement here. Obviously, its biggest feature is that it works on phpBB 3.2 and 3.3. This is the first of my extensions to be updated for phpBB 3.3.

It can be downloaded from my Smartfeed page or from the GitHub branch. If downloaded from GitHub, make sure to place it in the /ext/phpbbservices/smartfeed folder. Disable the extension first, removed the old files, then upload the new files and reenable the extension.

 

Filter by country extension, version 1.0.9-beta is now available for testing

At the end of December 2019, MaxMind, the publishers of the free GeoLite Country Code database used by this extension, required that users authenticate to download the database. So this version addresses this new requirement, but adds no new functionality.

You will have to get a license key to download the extension. The link you can use to get this key is in the extension’s settings page when you install this new version. You do not need to pay to get a license key. In fact, this version will only download the free database.

It will not work on phpBB 3.3, but I am told if you hack the ext.php file it will install and work correctly.

With this and my digests extension now fixed with their backlog of issues, I should be able to install a local PHP 7.4 development environment so I can test all my extensions for phpBB 3.3 and offer versions compatible with phpBB 3.3.

It can be downloaded from the extension’s page or from the GitHub branch. If downloaded from GitHub, make sure to place it an /ext/phpbbservices/filterbycountry directory.

phpBB 3.3 is released!

Two days ago, the phpBB group released its latest minor version of phpBB: 3.3, also known as Proteus. You can learn more about it on its launch page. To give you some perspective, phpBB 3.2, the last minor version, was released on December 9, 2016. So it’s been three years since the last minor release of phpBB.

I looked at a development version a couple of weeks back. So I was kind of taken by surprise by 3.3’s sudden release. Minor versions tend to introduce some new functionality, and Proteus does. It’s just that for most administrators and users, it won’t seem like that big a deal and things will look and behave pretty much the way they always have. As with phpBB 3.2’s introduction, most of its changes are covert, rather than overt. Unless you know what you are looking for, you won’t notice much.

New logo

One hard-to-miss feature, at least if you use the default proSilver style, is the phpBB logo is different. It’s now a Scalable Vector Graphic (SVG), which makes it look crisp and shiny in all resolutions, including retinal displays. It looks a tad bigger, but also more white and almost glossy. Also, the logo includes the words “forum software”, which is new. Previously, the logo was a transparent GIF and it said “Creating communities”.

New phpBB logo
New phpBB logo

Updating is getting easier

Updating phpBB is getting easier too. It won’t compare to updating WordPress, which takes place entirely behind the scenes and can be done with a single click. The exact mechanics of how it will work is unknown until 3.3.1 is released. But the launch page says:

“With our brand new installer updating will be easier than ever in phpBB 3.3! Upload a single folder to your board and all your files will automatically be replaced.”

This will be welcome because updating has always been a hassle. Over time, it may affect my income a bit since a lot of it comes from updates. I expect a lot of my customers will still want me to do this as a service.

You will still need to upload one file, an archive. I also expect there will be a number of other manual steps, because there will always be issues of overwriting custom changes to styles and extensions that may have issues. You will probably have to back up your styles and extensions folder manually before updating. Time will tell.

New PHP requirements

Proteus requires PHP 7.1.3 or higher, and cannot use a version of PHP greater than 7.4. So many administrators will have to upgrade PHP first, which may be an issue for those using PHP 5.4 and 5.6. They will finally have to take the plunge.

Most likely a lot of these boards will have an issue: they will need to edit their config.php file to tell phpBB to use mysqli drivers instead of mysql drivers. So far, fixing this issue has not been intuitive.

Overall, taking the plunge to PHP 7 is good: twice the performance compared with PHP 5 and phpBB can use many new features in PHP 7 too. I have noticed some extensions have issues with PHP 7, however, for example the AWS S3 extension.

Upgrading from phpBB 3.2

The upgrade process from 3.2 is pretty much unchanged from 3.1 to 3.2, and will be more manual in nature than the newer upgrade process. You can see the steps required here.

Improved Emoji support

From a user’s perspective, the exciting feature is likely to be increased Emoji support. Previously, only certain Emoji characters could be used, and only in post text. Now you can use virtually any Emoji character, and you can use them in topic titles too. However, you cannot use Emoji in the subject line of topic replies.

The Emoji in topic titles permission is enabled by default. If you want to disallow it, the easiest way is to change a user role, like Standard Features. ACP > Permissions > Permission roles > User roles > [Role name]. Click on the green wheel for the role. See illustration:

New emoji in topic titles permission
New emoji in topic titles permission

Over time, phpBB forums will look a lot more colorful and visual.

No support for IE before IE11

Also with Proteus, phpBB essentially gives up caring about Internet Explorer versions 7-10. It’s not that phpBB won’t render pages with these older browsers, but certain features won’t work or may behave quirkily. This is because to do fancier things, phpBB relies on a Javascript library called jQuery. It now uses a newer version of jQuery which is not compatible with these older browsers. The phpBB Group’s rationale is that since Microsoft won’t support old versions of Internet Explorer, they don’t have to either.

Some other features they are highlighting:

  • Clever quotes. Quotes can show a link to the post and post author. It can also show the date and time of the quoted post.
  • Improved reCAPTCHA. Previously only reCAPTCHA V2 Checkbox was allowed for a reCAPTCHA solution. Now you can use the Invisible reCAPTCHA V2. One consequence of this is that the V2 Checkbox is no longer supported, so as part of upgrading phpBB to 3.3 you should have to get a new set of reCAPTCHA keys from Google’s reCAPTCHA site that support this method, and plug them into the Spambot Countermeasures area in the ACP.
  • Notifications are supposed to be very fast now. The whole notifications process has been reengineered. It’s unclear if this means email notifications are sped up. I’m pretty sure they will go into phpBB’s mail queue like they do now, so your Email settings should apply.
  • FontAwesome improvements. In phpBB 3.2, phpBB supported a rather limited set of scalable FontAwesome characters. The number supported are now much larger, and they will all look fine on retinal displays.
  • Symfony 3.4. This is behind the scenes stuff, but phpBB 3.3 uses a newer version of the Symfony PHP libraries, including its heavily used template engine.
  • ACP Statistics screen is now responsive. As noted in my first look, the statistics panel in the ACP now splits statistics into two groups, which has the benefit of making the screen responsive. You can see the new look below:
New ACP Statistics screen
New ACP Statistics screen

Should you upgrade?

You probably don’t want to upgrade right away. This is because some of your extensions may not work and if you made changes to your style, those won’t carry over so they will need to be replicated.

However, the same day the phpBB Group also released phpBB 3.2.9, which brings over some of these features including Emoji support. You might want to update to that version for a few months until extensions and your style becomes compatible with phpBB 3.3.