Month: May 2017

Digests “error while creating image” fix

On: By: Mark

Many digest users have been having issues running digests, particularly versions 3.2.2 and 3.2.3. At the suggestion of a user I introduced a feature that allows month abbreviations and days of week to be language independent by using phpBB’s $user->format_date() function. Aside from most if not all of digests not going out for a particular hour, while you will see a “Starting digests mailer” entry in the admin log you won’t see an “Ending digests mailer”. In the phpBB error log you should see this:

Error while creating image
» Error in [ROOT]/phpbb/user.php on line 599: DateTime::setTimezone() expects parameter 1 to be DateTimeZone, null given

What is going on is that phpBB expects a PHP DateTimeZone object to be created and attached to the important $user object as $user->timezone. If it’s not there, the error will be triggered because the object doesn’t exist:

 $time->setTimezone($this->timezone);

In my testing, this happens when “Anonymous” (the guest account, or user_id = 1) is running digests. Essentially “Anonymous” is hitting the forum and triggering phpBB’s cron. phpBB’s cron may trigger the digest cron (if an hour has elapsed since digests were last run). Even though “Anonymous” has a timezone associated with the account (GMT) either the timezone object is not getting created or it is getting destroyed when it goes through the loop that sends out all digests for a given hour.

The solution is documented here and should be applied to versions 3.2.3 and 3.2.2 if that is installed. The instructions assume 3.2.3 is installed. Essentially before calling $user->format_date() it tests to see if the timezone object exists and if it doesn’t it creates it. There are three possible instances that could trigger this, so all should be fixed.

There is a general problem with the digests architecture. It extensively uses phpBB’s built in tools, such as this library, but also the templating system to create pretty digests, which assumes a live user is interacting with the forum. However, digests are often run by “Anonymous”, which is not quite a “full” user, or by a system cron. phpBB was clearly not designed to use these features (at least not very well) in “cron” mode. So as these problems are discovered, they must be found and fixed. And unfortunately these sorts of problems are devilishly hard to track down as they can’t be discovered interactively and only through tedious debugging of a sufficiently large forum by adding entries to the log. One of my clients essentially used his large forum to help me troubleshoot the issue and fix it.

The problem is not manifest when manually running the mailer. I can’t actually test digests from the command line (I need an email server in my development environment to do that) so I have to test in phpBB cron mode. A sufficiently large test database might have revealed this problem, but of course digests are highly dynamic so it’s hard to set up one of these.

Changes will get published to GitHub and this will probably be part of a new version, but not right away. I started reviewing the code to clean it up, and need to integrate these changes into that (and test it!) before releasing the next version.

Smartfeed 3.0.7 Released

On: By: Mark

Smartfeed 3.0.6 was not approved but the issues were quite minor. A review always takes time because there is a review queue, but in addition it took weeks before I could submit 3.0.6 because I had bundled SimplePie, and the library was so big that it couldn’t be uploaded. Eventually the webmaster fixed the problem and it could be reviewed.

Anyhow, 3.0.7 is now available. The only change in functionality is that Smartfeed no longer uses mcrypt (it is being deprecated) and uses openssl instead. openssl is almost always available so it will be a rare board that can’t support encrypted feeds.

Download here from GitHub or grab it off this site.

Here are a list of changes:

  • PHP’s openssl library is now used instead of mcrypt. As a result of the change, users may have to run the Smartfeed user interface to create a new URL. They will get a message in the feed to this effect if it occurs.
  • Fixed some links to use the helper class’s route method.
  • Replaced <br /> tags with <br> tags since HTML 5 is more flexible than XHTML which is no longer used.
  • Language strings use curly quotes.
  • Removed language variable that were no longer used.
  • Changed the URL to the Smartfeed page to prefix it with www.

Digests 3.2.3 (RC10) Released

On: By: Mark

In this version (compared with 3.2.2):

  • Improvement: $user->format_date() is used to create dates that show correct date abbreviations based on the language pack used. Thanks to Petr Hendl, the Czech digest extension translator, for finding this built-in approach. Consequently, I removed the dateFormatToStrftime() function.
  • Bug fix: Balance load previously ignored inactive users only. Now it also ignores bots, meaning normal users and founders are the only user types balanced.
  • <br /> tags changed to <br> tags, since HTML 5 is not particular like XHTML.
  • Improvement: make_tz_offset() checks for invalid user_timezone strings. If it is invalid, UTC is used for the timezone.
  • Bug fix: Fixed a bug in the validateDate() function. It was incorrectly finding every user_timezone string invalid.
  • Improvement: The digests cron calculates the date and hour digests were last run successfully, and uses this to determine if digests should be run. This should solve the potential hour creep problem and allow digests to go out for the hour on the hour even if the last time they were successfully sent out was not on the hour.
  • Bug fix: The cron includes includes/functions_content.php only if it is not already loaded.
  • Bug fix: Created a circuit breaker that prohibits a digest from going out as a duplicate if the timestamp for the time a user last received a digest indicates it is for the current date and hour.
  • The cron adds a language pack only if it has not already been loaded.
  • Improvement: Curly single and double quotes are now used in language strings.
  • All hours processed by the cron show successful completion, even though individual digests may not go out successfully. This keeps the cron from getting stuck indefinitely at a certain date and hour.

This version can be downloaded from GitHub or you can grab it off the Digests page.

Session hijacking: what’s (probably) going on

On: By: Mark

Over the last couple of months I’ve had a number of clients come to me because of mysterious things happening on their forums. Going to a forum they find that they are logged in as someone else and can see things they definitely should not see, such as private messages and forums they don’t have privileges to see. I’ve spent a lot of time trying to figure this out talking to client’s web host support teams and scouting phpbb.com for a solution.

The good news is that this is not due to some deficiencies in phpBB. The bad news is that this is due to the way your web host has configured their servers and it’s affecting phpBB.

phpBB is the #1 forum solution, with something like 70% of the market. But as a percentage of popular software installed on websites, phpBB is tiny, on about 1% of websites. What’s the 800 pound gorilla? It’s WordPress, which runs 27% of websites. So web hosts will meticulously tune their servers to optimize for WordPress, giving short shrift to much of the rest of the open source software out there. Most web hosts now say they are optimized for WordPress and market WordPress-specific hosting. phpBB is being left behind along with lots of other software. Because phpBB gets most of its content from a database to be presented on the fly, more than most open source solutions it is not amenable to static content.

The problem is most acute if you have Bluehost shared hosting. The underlying issue is some software called Varnish, more specifically Varnish HTTP Cache. Varnish helps dynamically driven sites perform more efficiently by caching content in your server’s virtual memory. Web hosts can make more money if they can get more utilization off one one web server. Varnish is one way they keep costs down as it allows them to stuff more websites on one machine.

Varnish is kind of pointless with phpBB since phpBB already has its own cache, which you can find in your forum’s cache folder. Essentially phpBB programs, templates, stylesheets and SQL calls are all compiled into .php programs in the cache folder so they can be executed more quickly. So it’s duplicitous but more importantly interferes with phpBB’s default behavior. So if you have the issue, contact your web host to find if they are using Varnish and if so have them turn it off. As for Bluehost, as of this writing they will tell you they can’t turn it off. You have shared hosting so one size fits all. They will however be happy to move you to their cloud product. Varnish is not installed there, so you won’t be affected. However you may have to pay a higher hosting fee.

It’s unclear if Varnish is the sole cause. Other potential problems may be due to Content Delivery Networks (CDNs). This is most typically CloudFlare, since it is bundled free by most web hosts. CDNs attempt to move content closer to the user by having it fetched from server farms geographically close to the site viewer, thus speeding up page load time. This is usually fine with phpBB since CDNs generally only store static files like images. So a CDN shouldn’t cause issues like this, but if you have a CDN you might want to disable it to see if the problem goes away. Note: the one time you do need to do something with your CDN is when you add a style or significantly change the look of your site. Then it’s a good idea to tell the CDN to delete all its cached content. Otherwise, the experience by end users might be mixed or odd.

I also suspect that ModSecurity may be causing issues like this, but I don’t have enough proof yet. If it is enabled, disabling ModSecurity may make your problem go away. As I blogged recently, disabling ModSecurity in general tends to solve a lot of weird phpBB issues, while it may introduce others by potentially making it easier for your site to propagate malware and viruses.

April 2017 work summary

On: By: Mark

This turned into being a pretty busy month here at phpBB Services. There’s no accounting for when work comes in the inbox, but the work is consistently upgrading forums for the most part. Also this month I released a new version of my digests extension and continued teaching a Javascript class at a local community college. Anyhow, here’s some of the paid work I did in April. As usual all client information has been anonymized.

  • I upgraded forum from phpBB 3.0.12 to 3.2.0 using default prosilver style. The web host first had to move everything to a new server that supported the newer version of PHP needed. PHP 7 had to be used as 5.6 was not an option on the new server. One ripple from this was that I had to change the forum’s config.php to change the database interface mechanism from mysql to mysqli for the installer update to work. This client also had my digests mod installed so I had to migrate it to the digests extension.
  • I changed some links on the navigation bar on an Absolution style on a phpBB 3.0.12 forum.
  • A client made mistakes upgrading his forum from phpBB 3.0.11 to 3.2.0. Some tables were missing from the database. The client eventually found a database backup file dated 4/2 that looked complete. Since the forum was large I had to do the work on my local machine. I loaded the database into my local database. The first upgrade I attempted was to phpBB 3.1.10. Then I did a second upgrade to phpBB 3.2.0. Since it could be done successfully, I moved everything back to the web server, loading the database with bigdump.php to avoid hitting potential resource limitations. There was a peculiar issue with cache directory (it could not be written to by programs), so I renamed it and created a new cache folder. Upon further inspection, we discovered the database was 5 years old, but the client finally found a complete and reasonably up to date extract of the database, so I converted and loaded that.
  • I was asked to rehost a forum. It turned out to be quite simple since the client provided an extract of the database, so I just had to move the files, load the database and change the config.php file. phpBB 3.2.0 was installed. When going into the ACP, it strangely redirected to website’s (not the forum’s) home page. The solution was to disable modSecurity.
  • A client has a MediaTemple virtual server. I kept getting noticed that various operating system packages needed updating. With the client’s permission, I updated 219 packages.
  • Continued work from the previous month with a user who had a custom content management system that previous developers had integrated into phpBB 3.0.7-PL1 forum. A particular user could not stay active. The content management system kept making him inactive. The only program I could find that would make him inactive was an integration with in their CMS. But it should never get triggered as his membership expiration date is in 2019. I checked all ban lists that would auto ban him but there was no apparent issues there. Customer wanted an estimate on how much it would cost to upgrade the forum, retrofitting his CMS for the latest version of phpBB. Spent 3 more on an analysis of the various options, none of which were cheap or easy.
  • I upgraded a forum from phpBB 3.0.11 to 3.2.0. I installed a proprietary Eles style from ThemeForest. The client has Hostgator hosting, so he had to have PHP upgraded first. After the upgrade, I had to delete a ton of old modules manually for mods that no longer existed. I installed new Statcounter code using an Eles widget and attached it to the footer. I provided guidance on updating domain information with the domain registrar and using HTTPS with the form. There may be additional work in few weeks. Some days later, a problem developed. Files with the mask autoload_* in /cache/production folder were getting removed causing sporadic errors. Hostgator fixed it, but not did not detail how they fixed it. I suspect modSecurity was doing this. At client’s request, I tested registration as a new user. The registration email went to my GMail spam folder. Suggested creating a board “from” email address under the forum’s domain. This worked and the registration email no longer went to a spam folder. Later the client purchased and integrated a security certificate, I added redirect code to the .htaccess file to force http traffic to use https. Later, client found he could not integrate the external image he wanted as a header background on the Eles style. I was able to grab the image, stored it in the images folder, and reference it correctly in the Eles style.
  • Troubleshooting. phpBB 3.1.6 wasn’t coming up. Looking at Bluehost logs suggested the issue was due to modSecurity being enabled. Adding a check to disable it in the .htaccess file but it didn’t solve that problem. I asked client to talk to Bluehost. I did download the software to check it with a 3.1.6 reference. There were no issues that suggested hacking. The next day client said now there were HTTP 500 errors. I investigated and found that the folder for the forum was missing most of the files! I uploaded what I had (it’s a good thing I keep backups before starting work!) from my backup as well as some missing folders like cache and store. The files folder looked intact, but images were missing, so there may be missing avatars. I changed folder and file permissions to standard. However, there were still HTTP 500 errors. Investigation showed PHP could not interpret namespaces, so PHP 5.2 was being used. Changing PHP in Bluehost did not affect the forum. So I edited the .htaccess file to force PHP 5.4 to be used and the forum came up. There were still issues of at least one image not appearing and messages saying I didn’t have permission to see certain images. Client’s phpBB account is missing these permissions, perhaps by design. Client needs to look it over and report back.
  • I installed my digests extension for an existing client on a forum he has on another domain. But I couldn’t delete Viglink from ACP screen. I tried uploading phpBB source files again but it made no difference. Suspected it was due to modSecurity. Eventually Dreamhost replied with instructions on how to deactivate it, so I deactivated it. That solved that problem. A new problem emerged: although the modules for adding extensions were installed, I could not see them in ACP > Customise! Sought help on phpbb.com. A row was missing from phpbb_acl_options table. Once added, I was able to install my digests extension.
  • A client could not login to his forum. Somehow phpBB 2 files were in his forum folder although the database said phpBB 3.1.9 was installed. Uploaded those software needed to run phpBB 3.1.9, pointed config.php to the correct database and it came up again. There may be files missing in files folder as nothing is there, but avatars are in images/avatars weren’t lost.
  • I converted a forum from phpBB 2.0.15 to 3.2.0. Initially there were issues because the web host control panel did not have a file manager. It was added by client. Mandarin Traditional Chinese language pack was installed. During the upgrade two issues of duplicate usernames were fixed. There was another issue that was due to some inconsistency between private messages and posts. I commented out the error message to get the upgrade to finish. I recreated the search index. I moved the old forum into old_forum directory. It is disabled and the server path was changed. Tens of thousands of images copied fine during the conversion. There was a Smartor gallery mod for phpBB 2.0 that was not migrated. This will be taken on as a separate project when a version of the phpBB Gallery extension for phpBB 3.2 is approved.
  • Going back to a client I helped back in February, there was a lingering issue with a login link not working due to weird hosting configuration the host would not change. The login box on the index worked, so I moved the login box to the top of the index page for visibility, and hid the login link that didn’t work. This is another reason to avoid web.com as a host.
  • Client had issues with session hijacking. I changed security settings to enable IP session comparison. In addition, Tapatalk seemed to be causing duplicate posts. Client deleted the Tapatalk extension files but did not delete the extension properly via the ACP. I deleted the remaining files then deleted obvious items for Tapatalk in the database in config, migrations and ext tables. I suggested disabling/removing extensions if problems recur. Not sure why but Bitninja was blocking certain IP addresses. False positives?
  • Troubleshooting. Newly registered users were going through moderation even though they were allowed to post immediately upon successful registration. Fixed by changing the newly registered user posting role to allow posting without approval. Approved 5 posts in the moderation queue and deleted 1 spam post reported in the moderation queue. Provided estimate for upgrade to 3.2.0. Later, some forum permissions did not allow registered users and newly registered users to post. I changed both groups to use standard access. I also upgraded the forum from phpBB 3.1.9 to 3.2.0, reapplying the logo and the Facebook button in the navigation bar. I also upgraded Tapatalk extension.
  • Error message accessing the ACP. Forum was running phpBB 3.0.4 and PHP 5.6. The issue was the newer version of PHP 5.6 was enabled without telling the client. Fixed issue by adding a line to config.php to turn off QA_HTML_COMPRESSION. The larger issue was the forum software was old so I upgraded the forum to phpBB 3.2.0. Used default prosilver style with upgrade, added the logo, and enabled the question on registration spam countermeasure.
  • I upgraded a forum from phpBB 3.0.11 to 3.2.0. First, I upgraded PHP from 5.2 to 5.6. After the upgrade, I manually removed modules for the phpBB Gallery and Advanced BBCode mods. Installed the green style and integrated old logo. Spoiler BBCode would not work properly until I installed the latest Advanced BBCode extension. I reenabled question on registration. I added back to navigation bar some external links, 5 total, that still work. Created old_files folder for files and directories that look like they aren’t used and can be deleted but client decided he didn’t need them. Issues with icons not appearing on the post window. Reuploading some style files solved the problem. Later, issue on registration that was caused by a column not having a default value. The table column was used by a phpBB mod no longer used. I deleted a lot of obsolete columns from the client’s table and removed a lot of unneeded tables used by old mods as well.