November 2018 work summary

This was a slow month for my business. It wasn’t a problem. I made my goal for the year at the end of October and was also teaching a class. Good to have some downtime. It allowed me to get out a new version of my digests extension.

But I did do some scattershot work here and there in November, updates mostly, including this work:

  • Upgraded a forum from phpBB 3.2.1 to 3.2.3. Initially I could not login to the Administration Control Panel. Changed cache folder permissions to make everything writeable and eventually was successful. I had to copy all the files from a version 3.2.3 reference to get the updater to work due to TWIG library errors that occurred when I tried to update the database. I reapplied the forum’s style changes for the logo and a PayPal button. I updated the Advanced BBCode extension from version 3.1.3. to 3.2.1. Changed all tables to use the InnoDB storage engine instead of the old MyISAM storage engine. There was an issue with the topics table. I had to remove a MyISAM full index before I could change its storage engine.
  • Additional work for a client I mentioned last month. Last month I completed a long upgrade of their forum from phpBB 3.0 with many modifications. In this latest work I changed the mailing list extension to use a new “Subscribers” group I created instead of Registered Users for sending topic and post notifications. This made it easier to opt out people who did not want these emails by simply removing them from the group. I created the new group then populated it with a SQL query by adding everyone in the Registers Users group to it. I provided instructions on how to remove people from this group. I noticed an inconsistency in phpbb_user_group table, removed lots of bogus rows where the user_id in the table did not exist in the phpbb_users table. I manually removed a bunch of people who did not want these notifications from the new group. A few days later, I refined the query to also filter out those where a column I created called user_email_all_posts was set to 0. I used this approach when a similar functionality was done on phpBB 3.0. All the notification issues seem to be fixed now and no latency has been reported due to emailing when creating new posts. Phew!
  • Troubleshooting. A catchable fatal error occurred on a client’s forum. It took a while to get access to the web host control panel due to the client’s illness. Once I had it, I manually purged the cache. This generated a different error. I changed PHP from 5.6 to 7.0 though made it go away. I changed permissions on the cache folder too. Later I updated the forum from phpBB 3.2.1 to 3.2.4. When I tried to run the update it said phpBB was not yet installed! I searched through the phpBB source code to see what triggered the error, eventually discovering it was because a .lock file was present in the cache folder. I removed it and the update was successful.
  • Updated forum from phpBB 3.2.3 to 3.2.4. No issues. Later, I installed the mChat extension on the forum.
  • Updated forum from phpBB 3.2.1 to 3.2.4. Updated the AllanStyle-SUBSILVER style that was the primary style, putting back the old logo. I removed the old security certificate, and installed a free Let’s Encrypt certificate. I changed the forum’s .htaccess file to redirect HTTP traffic to HTTPS. I discovered one issue on the index: insecure content was being served. This was due to http:// being in the user_avatar column of the phpbb_users table to serve avatar images. Used MySQL REPLACE to make these avatars to be served securely. In some cases there may be a blank space now because the server did not support HTTPS. I changed PHP from 5.6 to 7.2. This work inspired this blog post.
  • Updated a forum from phpBB 3.2.2 to 3.2.4. Upgraded the Cleantalk extension to the latest version. I changed config.php to use mysqli. This allowed the forum to work with PHP 7.1 (it was on PHP 5.5).

October 2018 work summary

I completed a major project this month, and that represented the bulk of my work. This nonprofit organization has been my clients for nine years now. Earlier in the year I convinced them they needed to move their forum off of phpBB 3.0 to the latest version. As they are surgeons it was hard to pin them down so it took months of sporadic back and forth. It’s nice to finally bring this home and collect a nice four figure paycheck for all the work. Their old forum had a number of modifications and code that I had inserted into the base code to do things like send an email notification with topic and post text to all members whenever a post was made. That won’t work with phpBB 3.2. It took a lot of experimentation to figure out something that was acceptable. Some of the things I did for them:

  • Used DavidIQ’s mailing list extension to handle the post and topic notifications with the post text. I had to tailor it to make it send emails to all registered users, rather than just one email address. I also had to change the template text to meet their requirements.
  • Installed five different styles and customized each to meet various styling requirements. I used the Quick style extension to give users an easy way to change styles on the navigation bar.
  • Figured out a way that users could upload and embed “web ready” videos without needing to use the Quicktime plugin. IMHO this is a serious deficiency of phpBB that it won’t use the <video> tag for these formats like .mp4 and .ogg. Basically this meant hacking the attachment_body.html template. I also had to create a MP4 attachment type, add it to the Flash group, enable the Flash group and take other formats out of the Flash group. I also had to enable the documents attachment group. This allows other video types to be uploaded, however they have to be downloaded to be actually played. The general idea is to encourage these surgeons to upload MP4 files only.
  • We quickly discovered that when MP4 videos were uploaded they were usually uncompressed. A 12 second video was 19.1MB! I found ffmeg, a command line utility that allows videos to be compressed, resized and moved from one format to another. I integrated ffmpeg into their virtual server and wrote a PHP script that once an hour looks for new videos. If it finds an uncompressed MP4, it compresses it using the “fast” method and updates the metadata for the file stored in the database. I used ffprobe to examine the video’s metadata, asked for it to output the data in a XML format, then used PHP’s DOM object to grab the information I needed. It took more than a day to write it, but it will save them lots of storage space as well as make videos faster to load and play. Given these videos are of surgery, they needed to be kept private so hosting on sites like YouTube was out of the question. The whole forum is tightly locked down to members only, and uses HTTPS to encrypt all communications.

There were lots of extensions installed as well in addition to other minor requirements addressed. Some extensions provided functionality identical to the mods they had, such as the Recent Topics extension. In a few cases they chose to drop the functionality of their old modification.

In any event, this project is largely closed although I expect a few bugs will manifest themselves in production use I will have to fix.

Other work in October:

  • I updated a phpBB from from version 3.2.2 to 3.2.3. Installed a beta Medals extension.
  • Spent two hours of labor on digest emailing issues. Since the hosting was GoDaddy, I pretty much knew what the problem was: GoDaddy blocks outgoing emails it thinks might be spam, and doesn’t bother to tell the client they are doing this. It basically takes a technical conversation with them and to get the client to certify that they really don’t have any inappropriate content or malware on their domains.
  • Spent time troubleshooting an issue on why no posts were being created on a forum. The basic issue was low board traffic. Also updated phpBB from version 3.2.2 to 3.2.3.
  • Answered some questions on user statistics and most users online. On the client’s old version of the forum, the number of users included a hack that added inactive users and bots. They really wanted that back so I changed /includes/acp/acp_main.php to include all types of users, and warned them this adds a complication with updates and upgrades.
  • Updated two domains for the same client from phpBB 3.2.2 to 3.2.3.
  • Client needed to change the domain registrar from GoDaddy to another registrar for two domains. Helped her with this obscure process.
  • Upgraded a vanilla prosilver style forum from phpBB 3.1.6 to 3.2.3. The upgrade was very slow and seemed to hang. I tried a few hours later and was able to get it to resume and finish. There was an issue related to moving to a new server and having PHP 7 installed, which triggered certain errors on phpBB 3.1. Replaced logo.
  • Upgraded another vanilla prosilver forum from phpBB 3.1.10 to 3.2.3. I changed the spambot countermeasure to reCaptcha V2. I added latest Cleantalk extension. I changed user registration settings to user gets email. Changed PHP to version 7.

September 2018 work summary

Things slowed down a bit in September, which was okay because I also started teaching a class this month.

I did have one large job at the start of the month that more than paid the rent. This was a commercial job and it involved moving a forum from phpBB 3.0.14 to 3.2.2. I was working as a subcontractor. The contractor’s client had a very particular look and feel to his forum and he wanted that to be retained as much as possible, so the challenge was to do this despite the fact that many modifications on the forum were not available as extensions. Obviously this meant making sure the style closely matched the phpBB 3.0 style. I did the prototyping work in a development environment. It took a long time to get things tweaked before I was given permission to move the changes into production. The first attempt at an upgrade of the database failed due to a resource timeout. The second attempt worked but took about six hours using command line interface. I installed the mChat, ReImg Imag Resizer, Google Analytics, Post Count Requirements and later NavBar search extensions. I set up Q&A spambot countermeasure. I removed dead modules and dead software including Tapatalk. The rest was styling work. I created a custom style based on Prosilver SE and then tried to emulate look of existing site. I added archives link to navigation bar, put the Twitter logo in the footer, and customized the footer to add link to site. Three templates were changed. Style customizations were placed in stylesheet.css. Later, there was four hours of additional styling work working from a punch list. One of the challenges was to do the work quickly. It was four days from start to finish. Given that this was a big forum, this was a moderate challenge.

Also in September:

  • I updated a forum from phpBB 3.2.1 to 3.2.2. There was a routing error caused by modSecurity being enabled. I disabled it and was able to post. Also upgraded PHP from 5.5 to 7.1 for about 50% faster performance for the client.
  • Similar to my last work for this client, I created MySQL triggers to subscribe people in the registered users and newly registered users groups to get topic and post notifications, but on a different domain.
  • I updated a forum from phpBB 3.1.4 to 3.2.3. (phpBB 3.2.3 was released in the middle of the month.) I installed an updated Aero style. I removed the Board3 portal extension. I installed updated extensions: Topic Preview and Notify Admin on Registration. I upped PHP version to 5.6 from 5.4. I changed the spambot countermeasure to use reCaptcha V2. I installed my digests extensions (version 3.2.8) and tested email using both a manual test and cron.
  • I updated phpBB from version 3.2.2 to 3.2.3 for a client using update files method. Updated five extensions that were out of date.
  • I updated another forum from phpBB 3.2.2 to 3.2.3. I updated the Tapatalk extension from 2.0.8 to 2.1.3. I had to reapply a style change to common.css.

With the release of phpBB 3.2.3, the phpBB group not only fixed some bugs that really needed fixing (including one that only allowed one attachment per post) and the migrator issues that gave me hell, but it also appears that the update program will now update any extensions that need updating, if the links work correctly. This is normally a good thing but if you wrote any custom changes to these extensions this might overwrite them, and there is no warning of this. You are not supposed to do this, but I have clients who have done this or asked me to do it for them. The update program really needs some sort of warning or a switch of some sort so these don’t happen automatically.

August 2018 work summary

I kept busy with phpBB work in August but fortunately not too busy. It was all just right for a change.

The big project was a Amazon Web Services (AWS) rehosting project that consumed four days after about two months of erratic testing and prototyping. I wrote about the project as it relates to AWS here. However, there were other details of wrapping up this project unrelated to AWS:

  • A heavily modified AllanStyle SUBSILVER style was used, with the colors tweaked to closely match the old style. During the prototyping there were all sorts of things that had to be just right because the community is very fussy and they wanted extremely high fidelity for what they used to have. This style is basically you only choice now if you like the boxy, full width styling of the old phpBB subsilver2 style. Members in a certain group had to have a different color number name and font size, so it took a lot of puzzling there to get something acceptable. Fine tuning the styles included moving the view topic author column from the right to the left, and getting the width of this column just right to match what they had. There were some peculiar requirements such as having the Administration Control Panel open in a separate tab and adding additional links to the navigation bar.
  • Six extensions were installed: Advertisement management, Pages, Media Embed, Smartfeed, Smilies Scroll and Topic Preview. There was quite a bit of work getting the advertisement management extension to serve ads correctly and to the correct groups.
  • I created a forum test environment and kept it around so subsequent changes could be tested there before being deployed to production.
  • Some BBCode was in the database for ranks messing up the display of ranks. These had to be removed from the database.
  • There was a peculiar program that returned one line of Javascript to another site that needed to be modified because the upgrade introduced line feeds into the output.
  • Figured out why hide my online status wasn’t working. It was a permissions issue.
  • There was also BBCode in the user_jabber column that could not be edited and had to be manually removed in the database. It messed up editing profiles. I discovered the Jabber module was not enabled, so I enabled it.
  • Changed the text in the email templates to match content that was in there before.
  • Later in the month there were more requested tweaks. The client wanted more ads to display on the view topic screen, every ten posts. This is not built into the Advertisement management extension. I used the group template variables extension to get the group information I needed and tweaked the view topic template to serve the ads unless users were in a particular group. Now they want another ad on the index between forums that I am working on.

Other work in August:

  • I upgraded a forum from phpBB 3.1.6. to 3.2.2. I reinstalled an updated Latte style. Four extensions had to be upgraded: Joined Date Format, Board Rules, Reimg Image Resizer and Topic Solved. I reapplied the background logo and footer links. There was an issue when posting. Email notifications did not go out because SSL check didn’t work, causing an error. Disabling it allowed posts to be saved but email notifications still did not go out. Client had to get a correct set of SMTP settings from their web host to fix the emailing issue.
  • Stood up a new forum and new domain for an existing client of nine years using a forum solution largely the same as their existing forum, Siteground hosting was selected on my recommendation. Installed phpBB 3.2.2, placed logo after resizing it, installed an orangish style to match the logo colors. Hid forum content from public. Disabled user registration. Installed ACP Add User extension. Installed and tested Tapatalk extension and access. Set up email account and forwarder to test email interface via SMTP. Set up SSL using free Let’s Encrypt certificates. Added Mailing List extension with same changes I made for their other site. Created a robots.txt file to deny search engines. Disabled contact form. Need to manually add users when deployed. Currently this project is not complete and is in user testing. As a result of testing, I applied a new style and made a number of fine styling changes, including some fine color changes. I also applied new specified fonts to forum name and site description. Installed media embed extension to embed a video.
  • Client reported this error: “Error: [phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4190: ob_start(): output handler ‘ob_gzhandler’ conflicts with ‘zlib output compression'”. Solved by changing PHP to 7.1, turning off the PHP ziplib extension, but needed to turn on mysqli extension because on this host it was not enabled by default. Later, it was discovered that some searches timed out after 30 seconds. I changed tables from MyISAM to InnoDB storage engines, but this did not solve the problem. I changed the PHP timeout to 60 seconds but the error still happened. I changed search index to MySQL FullIndex and it did not solve the problem. Eventually I determined the PHP mbstring extension needed to be enabled. I filed this as a documentation issue with the phpBB Group as enabling this PHP extension is not a system requirement.
  • Updated a forum from phpBB 3.2.1 to 3.2.2. Was getting messages including “Parameter must be an array or an object that implements Countable.” The update made the error go away.
  • Upgraded Tapatalk mod for a phpBB 3.0 forum to version 5.0.1 and made necessary file changes too. I removed the Tapatalk rebranding module because it was giving errors and is not needed.
  • Styling work to integrate site headers and footers onto phpBB 3.2.2 using prosilver on a test site. 3 templates modified. Stylesheet changes were added to stylesheet.css. This project is not quite complete yet, but nearly so.
  • Upgraded Tapatalk mod for phpBB 3.0 from version 4.9.8 to 5.0.0 for a client. I removed the Tapatalk rebranding module because it was giving errors and is not needed.

 

 

July 2018 work summary

Happily, there was a lot more work in July than there was in June. I am noticing that I tend to get called in for more complex projects and fewer simple projects. One effect this has is that these projects can span months. I worked on two projects like this during July that I do not detail below, but will when finally completed. They will have amount to nice sales when they finally cash in. Both projects are complex upgrades from phpBB 3.0 where there were many customizations by previous developers (including me). Moving them to phpBB 3.2 involves a lot of talking, compromise and prototyping. Both involve standing up test instances of their forums and iterating through them getting the style and features right. As for the other work I did in July:

  • Upgraded a forum from phpBB 3.0.12 to 3.2.2. I had to do it minus the tools I usually have. I had only SSH access. As such there were various technical issues. Trying to make a database backup took a long time in phpBB. But with root privileges I was able to do it very quickly. There were issues uploading files. I had to change the Unix group I was in to get the file permissions correct. I seemed to lose ability to create directories using Secure FTP. I worked closely with client who did some work and gave me additional privileges. There was an unexpected issue with /config directory that did not properly upload and a missing line in the config.php file. Th install folder was not recognized. I had to fix some database issues with upgrade, mostly done from the command line. I reapplied styling changes including logo, various buttons and some help links. Search indexes would not work (MySQL fulltext) so I created phpBB native index instead. I removed dead modules and updated spambot countermeasure to use reCaptcha V2.
  • Troubleshooting. The host set PHP 7 as the new default version of PHP, which caused issues bringing up the phpBB 3.0 forum. I could only revert back to PHP 5.6 on a phpBB 3.0.12 forum but it came up. Later, I upgraded the forum to phpBB 3.2.2 which does work with PHP 7. There were no issues. I placed the logo, changed the spambot countermeasure to reCaptcha V2, added a home link, and removed dead modules.
  • I created two MySQL database triggers to add forum notifications, similar to work for another client in May. This was more involved since 17 forum notifications were needed for each user in the group. I also wrote SQL to make the phpbb_forum_watch table contain rows for all existing users in the newly registered users and registered users groups, so the new triggers would work for people in these groups. I analyzed an issue with notification emails being checked by default. It looks like the two critical ones (new topics in subscribed forums, and replies to topics in subscribed topics) are set by default. I suggested not to turn additional email notifications on by default but it can be done with a trigger if necessary.
  • Fixed a digests issue on version 3.2.7 of my digests extension for a client. Digests were not going out reliably with their cron. A manual test worked fine. To fix cron job I removed /forum from the path since the forum is in a subdomain and added /usr/bin to the curl path.
  • Issue of FatalErrorException in container_builder.php line 146, which brought the forum down. /cache/development folder had 755 permissions. Removed it, changed config.php so phpBB development mode was disabled, cleared the cache manually, and the forum came back up.
  • I upgraded a forum from phpBB 3.0.5 to 3.2.2. The old style was not available, so a new style choice had to be made. The default prosilver style was used. There were some minor issues during upgrade that I had to puzzle through. I added the old logo. I recreated the MySQL full text search index. I added reCaptcha V2 spambot countermeasure and enabled it. I removed some dead modules including old AutoMOD modules. I added a home link.
  • Troubleshooting. There were issues with users not getting email notifications when registering or for forum/topic replies. First I advised setting email package size to 0. That did not seem to work, so I was asked to test it. I created an account and verified that both the registration email was received and forum notifications were received. It is now working correctly. Some permissions looked odd: newly registered could only reply to topics and registered users could not post in most forums.
  • Installed the mChat extension on a new forum with the Flat Style Blue style. Used cPanel to upload the extension. Later, I made text changes to mChat (changing its title to “Live Chat” among other changes) and removed credit lin,e substituting one provided by user. Some days later, I installed the forum icons extension and an archive of forum icons mentioned in the topic for the extension. The next day, the client requested some tutoring so I created a tutorial video using SnagIt. I demonstrated adding an extension, using cPanel, Filezilla, Dreamweaver, File Manager, making a sized and transparent logo and swapping the default logo, changing phpBB’s default text.
  • A site got hit by malware and was recovered. After recovery, I was asked to check the extensions and base code for malware. I upgraded my digests extension and the Cleantalk extension to the latest versions. I compared files with a 3.2.2 reference and found no malware. Later in the month, I removed unneeded phpBB software files, probably from previous versions, that remained in the forum folder. I also enabled the contact form check in Cleantalk because contact form spam was happening. Even later in the month, more spam was reported. Cleantalk extension was somehow disabled. I reenabled it. I removed spam users via Cleantalk once it was reenabled.
  • Troubleshooting. There were regular HTTP 500 errors accessing certain folders in the ACP. When I finally found the web server error log, the log indicated that the /cache/production/twig folder could not be written to. 755 permissions existed. I changed the permissions to 777 and deleted the files inside it that had 755 permissions and the issue was resolved.
  • Troubleshooting. The forum could not send mass emails. Since he has Siteground hosting, passed on what I learned when I set up my hosting: create an account on the domain and configure phpBB to use SMTP. It took a little back and forth with Siteground technical support but this approach eventually proved fruitful.
  • This job was a referral from a recent client. I was asked to straighten out an inconsistent .htaccess file for the site. Issue were http to https redirects weren’t happening, index.html page needed to be hidden if called directly and certain hotlinking needed to work right.
  • The Cleantalk extension was disabled, so spam started to appear, same issue as another client had this month. I updated the Cleantalk extension to the latest version and reenabled it. I found about 20 spam users that I deleted.
  • I installed the Detailed Online View extension, a development quality extension, after warning client it was a bad idea to use any extension that was not approved. It is accessed through the Who’s online link.

June 2018 work summary

June was a very slow month. This wasn’t bad as March was so crazy and I was teaching a class too. Summers are often slow around here as people have other things to do with their time.

I used the month to get out version 3.2.7 of my digests extension instead. I actually did quite a bit of analysis for clients, trying to work through the requirements of a job. A lot of my work these days involves very complex upgrades where many things have to factored and considered by the client before moving forward. I expect some of these will get authorized and worked on in July. Anyhow, it was nice to have a break. Overall though I’d prefer work to be more regular rather than either light or super heavy.

  • I upgraded the Cleantalk extension for a client to latest version (5.6.5) per client’s request. I also changed the spambot countermeasure from simple image to reCaptcha V2.
  • I upgraded a forum from phpBB 3.0.12 to 3.2.2. There were lots of issues associated with the client’s hosting. FTP did not work reliably (regular timeouts) meaning lots of files had to get requeued and resent. Upgrade was to place forum in a new /forum folder instead of root folder. There were issues upgrading on web server, so I converted database on my machine. I had to empty the migrations table to get it to work. One error discovered later by the upgrade: missing row in the phpbb_acl_options table meant that I could not install extensions. Once I added it, the manage extensions link appears in the Administration Control Panel. I installed the Absolution V3 style. Installed Thanks for the Post (RC) extension and Banner Scroll (RC) extensions. I had to move some files for the banner scroll extension into /forum/images folder. Lots of work tweaking the style. I abandoned provided site logo and reverted to earlier one as it worked better with the style. I tweaked the logo so it is kind of centered, hard to do with CSS blocks. Added G+, Facebook, YouTube and Twitter icons to navigation bar. Installed reCaptcha V2 as spambot countermeasure. Later: finally got SSH access and that allowed me to move files and images folder. I removed all other content in public_html directory.
  • Configured Cleantalk for a client with two similar websites. I used Cleantalk to remove spam accounts from both of their domains. Additional work troubleshooting an issue with moderation, not quite understood yet, want to know if it recurs. I removed items in moderation queue by approving each then deleting manually from the topic. There were still issues with Cleantalk extension being unable to communicate with Cleantalk server. Both allow_url_fopen and CURL are enabled in PHP. Probably an outgoing firewall issue. I asked client to talk to web host about the issue and provided a list of servers that should be allowed to communicate with. Later, I was asked to resolve signature issues on one domain for a particular issue. The role was not assigned for many forums for the person’s group and in the case of some forums the Registered Users group permission for signatures was set to NEVER meaning the special group’s permissions were being overwritten. Later, I disabled search on another of the client’s domains because he had integrated a Google Custom Search field.
  • I migrated a forum and also upgraded it. BitNami was installed (phpBB 3.2.2) on new host so rather than simply migrating to new host I converted the database on my machine instead. SSH was my only tool, so I used it to change and import the database from the command line and make file changes manually using vi. I installed seven additional language packs. I removed dead modules. I installed reCaptcha V2 spambot countermeasure. I placed the old logo on the default prosilver style. About a dozen files would not copy over, and caused Filezilla to crash. This client was kind enough to leave a comment on my home page expressing satisfaction with my work.
  • I upgraded a forum from phpBB 3.1.3 to 3.2.2 on a Windows server running IIS. I provided guidance on a Joomla issue which led the client to take care of that issue first with a Joomla expert. The upgrade went fine and I reapplied logo on the prosilver style. However mysterious issue with posting (no refresh screen after posting) remained. I eventually determined it was due to incorrect integration of email by the forum. When posts are made, post notifications are sent. This was causing the error. I suggested raising issue with web host or moving to Linux hosting. Did not install Social Media login extension, although we discussed this.

May 2018 work summary

Things were a bit slow in May, at least for the first two weeks. Curiously all my work was for existing clients. The theme of my work in May: getting rid of spam and preventing it.

  • Client #1
    • May 4. I helped fix an outgoing email issue on GoDaddy hosting. phpBB notifications were being blocked by GoDaddy from going out, something that you would never know if you don’t call their technical support line and press them. GoDaddy detected malware on the domain but after running a scan could not find any. They required that the customer certify that there was no malware on the site before unblocking outgoing email. In response I created an email forwarder for the domain and made this the board contact email. Not having an email address for outgoing mail associated with the domain is often the root of these problems. Customer called GoDaddy, they reenabled outgoing email and notifications started going out again.
    • May 12. Issues with spam posts on the forum despite using the reCaptcha V2 spambot countermeasure setting in phpBB. Customer was paying extra for a GoDaddy Securi service that automatically detected and removed suspected malware, but which removed some valid software introducing issues. Securi zeroed out some software files that kept me from logging in. After I eventually figured out the problem, I replaced the two files and I was able to log in. Earlier I had removed copies of mods in the store folder (used when the board was using phpBB 3.0 software) that were flagged by GoDaddy as malware. There were recent spam posts and it’s unknown how they happened. It’s possible the spammer solved the reCaptcha manually. I deleted about 150 inactive and likely spam registrations via SQL. I provided user with a list of posters since 4/1/18 so he could make a judgment as to whether these were spam registrations. I provided instructions on removing these users and their posts. I reinstalled Tapatalk since it was not working. I had to do it through File Manager since FTP wasn’t working. mChat extension wasn’t working either. I tried to reinstall that too but it didn’t work. I needed to reupload all phpBB software files since components were probably removed by Securi. Noticed navigation icons were removed too. Talking with GoDaddy they disabled the firewall and got me FTP access again. I re-uploaded folders where Securi had changed things. This seemed to fix remaining issues. After purging the cache, mChat started working and icons reappeared. Uploaded Cleantalk but not enabled. Providing instructions for configuring it if client wants to use it. Had GoDaddy disable firewall. Customer may want to stop paying for it.
  • Client #2
    • May 12. Since my last work for client they moved their forum to a dedicated server. Could not get in with FTP credentials. Used file manager instead. Installed release candidate of Thanks for the Posts extension. It will work with Tapatalk if the mobiquo directory is replaced. Did that and tested and it seemed okay. Installed a beta Modern Quote extension. Hid the portal page link because a sidebar would not show on right due to styling issues. Changed server settings, cookie settings and .htaccess file to accommodate HTTPS. There were some issues with getting cookie settings to work correctly. I also had to change some URLs in the posts table to https and fix some ads that were showing http in the URLs. To accommodate a Discord chat widget client wanted on the sidebar, I created a .css file it uses and linked it in in overall_header.html. I placed it at the bottom of the right sidebar in a custom widget in the Board3 portal, but there were connection issues since the service was not authorized that the client fixed. I fixed a phpBB Gallery integration issue. In the config table, the row containing phpbb_gallery_num_comments was blank when it needed an integer, trigger a SQL error. Set it to 1 since there was only 1 comment in the gallery so far. Fixed edit mchat post issue so an admin could edit the chat post due to white text on white background. Applied mChat patch to force the window to scroll to bottom in mChat. Fixed background and text color of Discord widget with an inline style to the widget since the board portal stylesheet was overruling it. Clock widget bottom part did not refresh on Chrome, but does on Firefox. Not sure it can be fixed (browser quirk) so client removed it. Checked background color issue for multi-quote. It’s black for me. Does not appear to be permissions based.
    • May 13 – Changed the background color for Gallery display, was white, commented out in CSS. User provided a watermark and I added that. Changed cookie name, forces a new login.
    • May 14 – Sorted through issue with Gallery watermarks (should be applied to subsequent images) and virtual URLs it creates. Disabled Image upload extension.
    • May 16 – Mostly answered new questions, changing some settings such as a PHP maximum upload size. Installed Search User Topics extension.
    • May 28 – Client asked me to update the version of Tapatalk to 2.1.10. However, it didn’t integrate well with version of mobiquo supporting the Thanks capability. To get things stable, we eventually reverted back to Tapatalk version 2.0.8.
  • Client #3 
    • May 16 – Time consuming phpBB upgrade on dreamhost.com virtual private server from phpBB 3.0.14 to 3.2.2. Over 1.2M posts, took 3 days! Running a database update quickly failed. Tried the command line interface instead. Even then it was excruciatingly slow. Broken SSH connection overnight meant a restart which bogged down. A chat with tech support finally revealed the upgrade slowness issue: insufficient RAM for the database. Bumped it to about 600MB and the conversion finally completed. Everything was straightforward thereafter. Used the black style which I changed to make red type rather than blue to match the look under phpBB 3.0. I replaced the logo. I added the latest Tapatalk extension and enabled it. I added Cleantalk extension and gave instructions on how to enable it with a key from the website. Turned on its spam firewall feature. I created a robots.txt file to deny all robots. I enabled reCaptcha V2 spambot countermeasure. Changed group color for administrators so they could be seen easily in styles.
    • May 17 – Additional work. MySQL too many connection errors kept happening. Spent time with tech support on this issue. MySQL settings cannot be changed for the VPS type of account. Some users were reporting bogus login errors. Disabled Cleantalk and think that will solve the issue until Cleantalk is properly set up. Added prosilver_se style for visually impaired users who can’t see dark style well. Installed Quick Style extension so styles could be easily changes. Changed blue text colors to white but cannot change color of icons since they are images — would need new set of icons colored correctly. Client wants to jump to the first unread post of a topic when clicking on topic icon. Found a hack for it but advised against it.
    • May 18. Investigated some new issues. The search index unavailable message when posting, but post was saved. Changed search index to phpBB Native. Installed Naruto shipuuden style and tweaked it to place logo and make it look right.
    • May 19 – Answered some questions. Installed Thanks for the Posts (Release Candidate) extension. I did more tweaking of newest style to make it look more uniform.
    • May 20 – Answered a lot of questions. Think the too many MySQL connections issue is either related to Tapatalk’s mobiquo directory or needing to change the storage engine for tables to InnoDB. Updated Cleantalk extension to latest minor version. Changed PHP from 5.6 to 7.0. Uploaded version of mobiquo for Thanks for the Post extension.
    • May 21 – Installed ShareOn extension. I found a way to get to MySQL prompt for potential change of tables storage engines to InnoDB. Changed a lot of forum permissions. Explicitly gave Thanks for the posts permissions where necessary. Verified it worked in Tapatalk with the replacement mobiquo folder they provided.
    • May 22 – I changed all table storage engines from MyISAM to InnoDB. The search wordmatch table took about six hours to convert. This resolved the MySQL too many connections problem, probably because tables were being locked and unlocked more efficiently and at the row level instead of the whole table level. I also installed the Media Embed plugin. I wrote a little blurb of instructions for client’s users and provided a screenshot for using the Media Embed plugin.
    • May 24 – I installed Advanced BBCode Box 3 extension to make it easier to embed videos.
  • Client #4. To address a persistent spam problem, I installed the latest version of Cleantalk and changed spambot countermeasure to reCaptcha V2.
  • Client #5. To address another spam problem, I installed Cleantalk extension on two domains.
  • Client #6. Client had a major spam attack with thousands of spam posts and spam registration. Configured reCaptcha V2 on phpBB 3.2.1. Then installed Cleantalk extension. However, there were issues while removing spam with Cleantalk, first hosting timeouts, then duplicate rows in phpbb_notification_types table. Emptying this table eventually worked. Removing spam was a combination of some SQL on the backend plus Cleantalk extension.

April 2018 work summary

In contrast to March, April started out busy and ended up slower. This is in part because much of my March work continued into April. Anyhow, here’s what I worked on for clients in April. As always, I do not identify my clients by name or domain unless they specifically allow it:

  • I spent a lot of time continuing work for a client I started working for in March. Their WordPress site integrates phpBB into a separate area.
    • First I completed the move of the forum from phpBB 3.0 to 3.2. The hard part here was making changes to the phpBB style to integrate the overlaying WordPress theme in the headers and footers, as there were many CSS class and HTML ID conflicts.
    • As part of the upgrade, the client took the plunge and moved the site from standalone web pages to WordPress. A lot of scripts written by someone else to monetize particular types of posts broke because of changes to phpBB’s API. These old programs were “tightly coupled”, to use a software engineering term. Basically as these broke I fixed them by updating the phpBB code inside these programs. Most of this work involved changing $_SERVER, $_GET and $_POST statements to use phpBB’s request object instead. 
    • With the new WordPress framework, some of the old content appearing in sidebars needed to move into WordPress widgets in the sidebars of the theme. We’re taking this piecemeal, learning as we go along. A better “loosely coupled” way to get phpBB content in WordPress is to use phpBB feeds. phpBB comes with a built in Atom feed that needs to only be enabled. But it doesn’t do everything to retrieve relevant posts. WordPress has a feed widget and there are also a number of other widgets available as plugins that do a fancier job of serving feed content from Atom or RSS feeds inside of WordPress. In the case of my client, the sidebar needed to show recently posted topics in the forum. I eventually decided I needed something beyond phpBB’s Atom feed, so I chose my own Smartfeed extension instead. However, it didn’t return quite what the client wanted as he only wanted “fresh” topics from a certain forum, and only the latest post from these topics. I was able to tweak my Smartfeed extension to do this, and this inspired me to work on a new release of Smartfeed to add these features. Once deployed, I will update the version of Smartfeed I installed for this client to use it instead.
    • I fixed an issue with moderator permissions with the forum by adding these moderators to a list of global moderators.
    • Installed a SiteMap extension so posts were better available to search engines.
  • Upgraded a forum from phpBB 3.2.0 to phpBB 3.2.2. Replaced the logo in the original dimensions. Upgraded Cleantalk to latest version on GitHub. Enabled the spam firewall feature of Cleantalk. Installed announcement on index extension for 3.2 (not official). Fixed issue with wrong reCaptcha. Checked global moderator privileges and they were okay. Administrator moderator role was changed to Full Moderator.
  • Upgraded Tapatalk extension to version 2.0.8. This also meant removing the mobique directory and reuploading that.
  • Changed the people who have moderator privileges for a client.
  • This work was a bit different. I created two MySQL triggers so if a user is added to a particular group by an administrator they are subscribed to receive notifications if new topics are created in a forum. I also created a trigger to remove the forum notification if they are removed from the group.
  • A client needed URLs embedded in the post text changed as the image hosting service has moved it to a new domains (postimg.cc and postimages.cc). It’s fairly simple to do once I finally figured out about the MySQL replace function. I also installed the Member Profile Views extension (1.0.3) for the client. Later, I updated the Cleantalk extension to the latest version. While I was in there removed 3 dead extensions from the extensions table that were showing as ugly error messages on the Install extension screen. I also help create a new BBCode to serve embedded videos from a site and updated the latest version of phpBB Gallery from the GitHub master: version 3.2.2.
  • Upgraded a forum from phpBB 3.0.11 to phpBB 3.2.2. Installed Allan Style Subsilver. Placed a new logo. Installed Advertisement Management extension and placed two old ads, changing the image URL a bit. The forum has 200,000 posts. One minor issue during upgrade: I had to make active the prosilver style for it to complete.
  • Completed a messy upgrade from phpBB 3.0.12 to 3.2.2 on GoDaddy hosting. The file folder was missing and was apparently dropped a month or so back when user went over quota, and was not recoverable. This meant lots of images embedded in posts would not show. The user has to live with this. I had to move the database to my machine to convert it, due to severe resource limitations on GoDaddy. The phpBB native search index creation did not work, so I created a MySQL Full Index one instead. mChat was previously installed as a mod. I had to manually remove all traces of it to get the mChat extension to install. Many tables were missing primary keys, indexes and automatic increment settings. I eventually figured out this was an issue with bigdump.php, a staggered MySQL importer I used, as it nears the end of a database extract file sometimes and hangs. These are stored at the bottom of the file and it never said it completed. It took an hour or so to create the missing indexes and primary keys. I did not move over tables for the search index, topic tracking information, the session information or the logs due to size issues. I could not find a way to remove spam posts easily but did remove old inactive users. Installed reCaptcha V2. Installed Tapatalk, Advertisement Management (to place ads at top and bottom), ShareOn, mChat (after cleanup) and OneAll Social Login extensions. Adjusted forum permissions so newly registered users could post without permission and gave full access permissions to all forums for all major groups.
  • As with the first client, I changed some URLs in the post text to use postimg.cc and postimages.cc. This one though was a bit different because periods were stored as HTML entities in the database. I also spent an hour trying to make the upload attachments feature work since he did not want to use external services anymore for posting images and he had plenty of space. There were lots of permissions issues to puzzle through. I had to change forum access privileges for registered users to full access. I also had to set the upload permission for the registered users to allow it and change the quota for uploaded images from just 1MB to 10GB.

February 2018 work summary

The nature of my business seems to be that it the work is few and far between or it all comes in at once. The first half of the month was mostly quiet. The second half was mostly busy … too busy. Anyhow, here’s a summary of my work in February. As always, I am not identifying my clients:

  • A lot of work here for one client. Basically, the client wanted a whole lot of changes to his forum. The endeavor was sort of successful.
    • I installed a development radio extension. This puts a player on the screen that plays music from a playlist for all members on the forum. I had to tweak the composer.json file to get it to install. To get the radio player images to appear on the portal, I had to hack one of the programs for the extension to use an image path that starts with / or they would not appear on the portal.
    • I tried to install Tapatalk a number of times but to no avail. I kept getting an installer error creating the Tapatalk tables. Upgrading phpBB didn’t work. So I tried installing an earlier version of Tapatalk. That didn’t install either. Since the issue was with the Tapatalk table not being created, I installed Tapatalk on a test environment, got the needed SQL there using phpMyAdmin and recreated it manually on the client’s site in phpMyAdmin. I changed the phpbb_ext table to make it active and cleared the cache. I could then see it in the Extensions tab and could enter the key.
    • Created an Apache redirect so non-www URLs went to www instead. Lines were added to .htaccess file.
    • Changed the color of the text in mChat when editing chat text. It was white on white, made it black on white.
    • Added a favicon for the site
    • Installed an extension to open external links in new tabs
    • Installed the Profile Flair extension. This allows badges to be created.
    • Installed the YouTube Video Gallery extension. It requires creating an application in Google console to enable, which the client did.
    • Updated phpBB 3.2.1 to 3.2.2. Applied three manual changes afterward, bugs that were subsequently found in the release.
    • Edited some colors on the portal sidebar that kept text from being seen.
    • Fixed a resizing issue with embedded videos on mobile devices by adding max-width:100% and max-width:100% CSS directives for the object tag.
    • Made more tweaks to colors for the “time” line near the navigation bar, made better CSS selectors for poll links in the portal and moved vote buttons in the portal more to the right, along with vote totals.
    • Radio extension turned out not to be viable. So I disabled it. I tried to place the player manually. I tried a number of approaches, once I figured out how to write the Javascript correctly. The easiest way was to turn it into an ad in the Advertisement Management extension and place it above the navigation bar.
    • More trouble with Tapatalk when the client tried to add a permanent Tapatalk key. Again the extension failed in an ugly manner and again the issue was some sort of database privilege that should have been there was not, probably for creating new objects like tables, columns and indexes. I commented out some code in the extension to get it to come up so the board was usable, but feels jury rigged. Installed some smilies that client needs to configure. The client is getting with his web host to try to solve the underlying issues of database privileges needed to create new tables that seems to be the issue with Tapatalk.
  • I upgraded forum from phpBB 3.0.10 to 3.2.2. I applied patches to 3.2.2 after the upgrade. There were two issues during the upgrade. The first (duplicate row in config table) went away by running the upgrade again. The second required using phpMyAdmin to edit the prosilver style and make it active. Then the upgraded completed. I removed some dead modules carried over from phpBB 3.0. I created keys on reCaptcha site and entered them into reCaptcha spambot settings, but they were not recognized. I eventually discovered if I requested invisible reCaptcha those keys would work. There was a similar issue with a Cleantalk authorization key. The client had to generate a key on Cleantalk site and he installed it himself. I also installed the Advanced BBCode extension. I noted an email error I encountered with the upgrade. It may be an issue with integration with email. Later that day when the client removed my FTP account (since it gave access to the web root) it also removed everything in the public_html folder! So his whole web site including the forum was gone! The web host had to recover it. The next day I reapplied the file changes for the upgrade and uploaded the programs for the extensions that were installed. The database was still good so I didn’t need to run the upgrade program again. The client asked me to look into errors in some 3rd party software tied to phpBB in his members folder. It asked for PHP 5.4. I changed his .htaccess file but that exposed an issue that the code was using super globals, so it didn’t work. I suggested contacting the original author and having him update the program to use the new phpBB 3.2 libraries. Client eventually figured it out himself.
  • Lots of work for this client too:
    • I converted a forum from phpBB 2.0.23 to 3.2.2. About 200,000 posts with about 7000 attachments. The conversion went smoothly. Allen Subsilver style installed. I replaced the logo. I recreated search index. I moved the old forum into old_forum folder where it exists as an archive.
    • Later I installed a Board Rules extension and Custom Registration Agreement extension.
    • Customer discovered a weird problem. When using the search index, sometimes a XML parsing error occurred. After lots of puzzling it through I realized I had seen this before. Non UTF-8 characters (probably belonging to the Windows character set) were in post_text column and other columns. The parser could not parse these characters because they were not translatable into UTF-8. I fixed it with SQL using MySQL CAST and CONVERT functions on all columns in tables where users type in text.
  • Removed and re-place some ads in overall_header.html for a phpBB 3.0 forum
  • I got called in as a consultant for a huge forum, the biggest I’ve ever seen: 14M posts! That’s about 4 times as many posts as are on phpbb.com’s forums which have been running since 2001! What a strange issue they were having! For about 48 hours they could operate as normal, then the database would freeze updating num_posts in the phpbb_config table. All they could do is recover the database and start again, then 48 hours later it would happen again. They were very frustrated. The error message suggested that some transaction was not getting completed. In addition, the value for num_posts in the phpbb_config was not consistent with the actual number of posts. On top of all this, this phpBB 3.0 forum has a site Wiki outside of phpBB, and the Wiki made posts to their forum using a MySQL stored procedure. What to do? It took hours to get command line access just so I could look around. Once inside MySQL I looked at the tables. Most were using the MyISAM storage engine, but the phpbb_config table where the problem was happening was using the InnoDB storage engine. If I changed its storage engine to MyISAM, would the problem go away? I did and the problem went away! Oh, I also wrote some SQL so num_posts matched the actual number of posts in the phpbb_posts table.
  • Upgraded a forum from phpBB 3.2.1 to 3.2.2. No surprises.
  • Updated a forum from phpBB 3.2.0 to 3.2.2. A direct 3.2.0 to 3.2.2 update would not work and caused Symfony errors. After much trial and error an upgrade to 3.2.1 then to 3.2.2 worked. I reapplied the logo and styles. I was asked to set permissions so users could edit their own posts and to fix an issue with their posts not going through moderation. It took some puzzling to figure this out. Apparently when a new user registers, they are placed in both the Newly Registered Users group and the Registered Users group. Unless permissions for Newly Registered Users are set to Never the privileges for Registered Users, if more generous, will take precedence. So for Newly Registered Users, I added Never permissions for seeing forums they weren’t supposed to see. Although newly registered users were required to have one post before they were considered registered users, posts are only moderated if the moderation permission is enabled, which it was not. So I turned on these permissions for this group.

January 2018 work summary

I didn’t do much work in January, mainly because I spent most of it on vacation: 19 days in total! It was a great vacation: a 16 day circuit of the Panama Canal on Holland America, from Fort Lauderdale to San Diego with lots of ports of call. In addition we stayed an extra day in San Diego to see its infamous zoo! I do have a few buns in the oven at the moment including a large project for a large client that I am starting work on that may take a month or two.

During my vacation my site’s certificate expired. I tried to fix the issue remotely when I had Internet, but just couldn’t do it with my Windows laptop. That may have cost me some business. It’s all fixed now.

On the infrastructure side, I upgraded my iMac to use a solid state drive. Things are so much spiffier now!

Before and after my vacation though I did squeeze in some jobs:

  • A client got rid of his forum due to low traffic but wanted me to do some WordPress work for him. He was been having WordPress security issues. I examined his WordPress .htaccess file. I found no issues there but redundant duplicate code was removed. I explained how the .htaccess file works and provided advice on updating, finding, fixing and proactively handling security for WordPress including Wordfence and UpdraftPlug plugins.
  • Two forums were upgraded for the same client, both from phpBB 3.0.12 to 3.2.1. Since the forums caters to German speakers, with the upgrade I installed two German language packs (casual and formal honorifics). On the first forum, I installed the prosilver_se style. I installed the Navbar search extension thinking it would help with styling but eventually disabled it as it was not needed. I changed style a bit to somewhat match old logo and background, experimenting with CSS until I found something that fit. Since the user could not login, I created a new account (after enabling registration) and gave myself founder privileges in the database. The next day I upgraded the second forum. I integrating the old logo but it did not look right with the new style. I tried changing the background color for the style but it still looked off. However, it was acceptable to the client. I then installed version 3.2.6 of my digest extension, and set up and tested the system cron job to ensure digests went out in the hour requested. I used curl and set the cron to run at top of the hour. I resolved a permissions issue that allowed any registered user to be a global moderator. There were some hosting issues: I could not find a link to cron in the web host control panel until the customer added it to my account. The next day I fixed a database issue: changed default for user_topic_sortby_type from m to t. This was apparently changed by some phpBB 3.0 mod. I updated the database to change this value to t for all users.
  • Troubleshooting. A forum gave a “is_dynamic missing” error message, which was actually a row in the phpbb config table. It took a bit of puzzling but it turned out that the forum’s config.php file was pointing to a phpBB2 database! No wonder the row was missing! I searched and found another database that looked like it had the tables for phpBB 3. But it was missing the last 5 tables. I found a copy of the database in the store folder dated 12-18-17. With customer’s permission, I reloaded that database. Bringing up board, the phpbb_login_attempts table was missing so I created it manually. I could not login with customer’s credentials so I created an account for myself and gave myself founder privileges via the database. The subsilver2 style was misidentified as Red Silver. I renamed it to subsilver2 in the database and enabled the Red Silver style. This fixed the styling issue and the site’s logo appeared in the header like before.
  • I upgraded a forum from phpBB 3.2.1 to 3.2.2. An issue with quoted characters repeating themselves horizontally and vertically, overwriting the quote, was solved with upgrade. Replicated logo in header and changes to footer. This was my first attempt to update to phpBB 3.2.2. I quickly learned that (a) the incremental update still does not work reliably and (b) you need to apply some patches.¬†Other than that (to quote Monty Python), phpBB was perfectly fine!
  • I upgraded another forum from phpBB 3.2.1 to 3.2.2. Database updater failed due to timeout. The real issue were some bugs in 3.2.2 and had to apply some patches to complete the upgrade. I reapplied the logo. The search index erroneously pointed to nonexistent MySQL index. Pointed it to phpBB native and message went away.