Month: November 2020

Delphi forums to phpBB – Part Two

On: By: Mark

In part one I looked at the technical approach I used for this unusual Delphi forums to phpBB conversion project. In the interim I have delivered something to the customer based on a somewhat dated extract they provided from their Delphi forums.

One thing I didn’t realize until recently was that moving their forums into phpBB was actually an interim state. The goal is to move it into bbPress, a WordPress plugin. Apparently bbPress can import from phpBB, but clearly not Delphi forums.

So it’s off to another contractor for that step. They might find they lose too much functionality moving to BBPress. phpBB after all has twenty years to mature and it has more functionality than any other forum solution, with one major exception in that it can’t do threads inside topics. Maybe that will show up in phpBB 4.0. If they lose too much functionality, they may keep it in phpBB, but they will need two sets of credentials because, in general, WordPress and phpBB don’t mix.

Some of the things I have learned and discovered since my last post:

  • Their guy who provided a YAML extract made a GitHub project out of it, so feel free to use if it you dare.
  • His parser though wasn’t perfect. It may be imperfections in Delphi’s software at fault. As noted in the last post, a few posts wouldn’t parse.
  • Downloading the attachments was challenging. Delphi does its best to not let you download them outside of its system. I ended up using curl and passing some authorization cookies in curl to download them successfully. Once downloaded, they then needed to be copied into a files folder so that phpBB could find them.
  • I had to write many PHP scripts to do the work and execute them in a defined sequence. Among these scripts were:
    • One that calculated the attachment size and placed it in an attachments table
    • A master script I called load_threads.php that did most of the initial work for reading the .yaml files and placing content into a set of denormalized tables that was used to store the posts and attachments data called phpbb_posts_load and phpbb_attachments_load. Subsequent scripts moved the data into phpBB 2 database tables after doing a lot of data jiggering.
    • Not all attachments had file suffixes, so I had to figure out what kind of file types they actually were and rename them accordingly. The Unix file command has a -mime_type argument that helped a lot.
    • One to load the users by examining user .yaml files provided in the extract.
    • One to resize images so they would not exceed a maximum height or width.
  • I also kept a set of notes documenting the sequence and issues discovered, and how to check the results. At points I had to move tables to my Raspberry Pi, which has PHP 5.6 installed, which can run the old phpBB 2 software, do some clean up there, then move the phpBB 2 database back to my machine so I could convert it to phpBB 3.3.
  • I created some pretty exotic SQL to do things like update the first and last posts (and posters) for topics, set the post count correct for topics.

In short, it turned out to be pretty tedious and a learning experience, but is definitely doable with the right skills.

phpBB 3.3.2 released, phpBB 3.2.11 also released

On: By: Mark

The phpBB group released version 3.3.2 yesterday. Since one security issue was fixed, an update to that version is a good idea.

I have updated the Do I need to update? page to indicate the tables that are changed in this update. If updating from phpBB 3.3.1, you don’t need to backup any other tables than those shown for the release on this page.

You can see the announcement and get a summary if interested.

phpBB 3.2.11 was also released and contains the security bug fix in 3.3.2, but which was present in 3.2. It’s better to migrate to phpBB 3.3 if you are on phpBB 3.2, but if you can’t you can at least address this security issue.

October 2020 work summary

On: By: Mark

October kept me real busy with phpBB work. But almost all of it was for two longstanding jobs. Other work coming in was rather small and ancillary.

I recently created this post describing my work on a Delphi Forums to phpBB conversion. I’ll add another post soon discussing the work I’ve done since this post. It amounted to eighteen hours of work during October.

Then there was the work for a large commercial customer that’s been underway since February. Most of my time was spent on this project. In October the focus is to figure out how to replace their existing proprietary OAuth interface (on a phpBB 3.0 board) with an extension. To say the least it’s a learning experience! I spent 46 hours on this in October and we’re not out of the woods yet by any means. My first thought was to update and inject their old phpBB 3.0 code where possible in 3.3. But the client nixed that approach as he wanted a clean, best practices approach. So we ended up hacking the Bitly OAuth authentication mechanism to connect to his authentication server instead. That in itself was a major time consuming project, but it was successful. The challenge now is to make it work through an extension. phpBB kind of gets in the way because it has no method to allow users to enter OAuth ID and secrets for a new service in the ACP authentication interface. So these keys must be entered in a separate page for the extension. There are a few events I can use to tie into if needed, but the client really wants to avoid using these if possible. Anyhow, it remains a challenge and requires a lot of adding debug statements to phpBB’s base code to figure out what’s going on. I think I may be close to getting a solution, but this is very advanced phpBB stuff. If worst comes to worst, I’ll hand it over to a more experienced phpBB developer.

Other work in October:

  • I moved a client frustrated with his hosting to Siteground. The client had moved the board’s files to the new hosting, but not database. So I had to do that and create a database and database user for the board and import the database. I then had to fix the config.php file to point to the newly loaded database. I then upgraded from the board from phpBB 3.2.10 to phpBB 3.3.1. I changed the spambot countermeasure to reCaptcha V3. I then went into the client’s domain registrar and pointed nameservers to Siteground.
  • Updated a board from phpBB 3.3.0 to 3.3.1. The client tried to use Automatic Update package but it screwed things up again. There were lots of FTP errors uploading and downloading to this particular host, so there were multiple attempts to move them all successfully before I could do the update.
  • A client reported issues with my digests mod on his phpBB 3.0 board. I verified Digests 2.2.27 is correctly installed. Client also complained of other issues when trying to post text and private messages. These appear to be due to using Kanji characters in UTF-8 text fields where the collation is wrong.
  • Certain Tapatalk servers could not be reached, so the Tapatalk extension could not be used by board users. The issue turned out to be on Tapatalk’s side since I got errors trying to use curl to contact the server. I complained to Tapatalk and about a day later the issue went away by itself. Later I upgraded Plesk to Obsidian 18.0.31 and updated 9 packages on their server.
  • Upgraded a board from phpBB 3.0.11 to phpBB 3.3.1. No issues occurred during the upgrade. I just had to stretch and place a logo. The board has 48,000 posts. Used the generic prosilver style. I disabled the contact form and installed the reCaptcha V3 spambot countermeasure. Created a home link on the breadcrumb menu.
  • Upgraded a board from phpBB 3.0.14 to phpBB 3.3.1. I couldn’t get FTP to work so did all file transfers using the file manager. I had to comment out last line of .htaccess file to disable loading PHP 5.6. Installed the mentions extension and tested it — seemed to work. Some 777 folders had 700 permissions — warned client about this. Disabled the contact form. Installed reCAPTCHA V3 spambot countermeasure. Deleted about six dead extension links on the system tab.
  • Two issues were reported by a client on their phpBB 3.2 board. First, the posting body text box was tiny. Underlying issue was PHP 7.4 was automatically enabled on phpBB 3.2.2. Changed that in cPanel and the issue went away. Second issue was that forum rules were showing even if you are logged in. Since these are properties of each forum, there is no way to get around this. As I was doing the work, a web hosting Securi security package whitelisted my IP and kept me from doing any work. I disabled ModSecurity but couldn’t disable Securi. Client did that for me. Client said Securi was disabled. I found a Securi configuration file with board database credentials that I commented out and things became much more stable. I was then able to upgrade phpBB to version 3.3.1. Due to timeouts on the hosting, I had to upgrade the board on my machine and upload it. There were lots of issues after the upgrade. Certain ACP tabs would not come up. View topic and posting worked sporadically, otherwise white screens occurred. After a lot of analysis, it looked like some existing Forumhulp extensions did not work reliably on phpBB 3.3 and hadn’t been updated by the extension author. Tried some hacks hoping they would work but the problems persisted so I left them all disabled. Suggested leaving ForumHulp extensions and Securi disabled.
  • Fixed a couple WordPress issues for a client, mainly with the StopBadBots plugin by uploading the most recent version. Also fixed a “Constant WP_MEMORY_LIMIT already defined” notice by removing it from the bottom of the wp-config.php file. It seemed to be cached but eventually went away. Inquired about the plugin to backup to Google Drive. Purchased extra Google Drive storage, installed the UpdraftPlus plugin and configured it to work with Google Drive. Started manual backup. Provided link to phpBB Auto DB backup extension. With UpdraftPlus premium, the phpBB board directory could also be backed up.
  • Upgraded a board from phpBB 3.2.9 to 3.3.1. PHP was already set to version 7.3 but no issues using phpBB were reported. (phpBB 3.2 works on versions of PHP through version 7.2) There was an issue of no attachments allowed in private messaging. No attachment group was allowed for PMs. Change the attachment groups for private messaging to match what was used for various forums. I also updated Cleantalk, Digests and Stop Forum Spam extensions.
  • Issued a new edition of my book and move the publisher to Lulu.com. Details here.