Digests 3.2.2-RC9 Released

I noticed an obscure but critical bug in Edit Subscribers that misreported the hour the digest was sent in some cases, so a new release was warranted so quickly after the last one. In addition, the display hour in Edit Subscribers show shows a formatted hour. For example, it used to show 14, now it will show 2 PM if user_dateformat requests hours to show as AM or PM.

That’s pretty much it, except for updating the version number and giving some better names to variables.

To get the latest version, go to the digests page.

Digests 3.2.1-RC8 released

There’s lots of new stuff in this release and lots of bugs fixed. You can download it from GitHub here or from this site here. This version works on phpBB 3.2 Rhea and will not work on phpBB 3.1.

To update, make sure you follow the standard procedures. First disable the last version. If you delete data you will lose all digest subscriptions, so it’s not recommended. Delete the old files in /ext/phpbbservices/digests and replace it with the new files. Then enable the extension again.

Big thanks to all who reported bugs and asked for new features. I would be especially lax if I did not mention my French language translator Bonnaphil who asked lots of questions, did lots of testing and pushed me to deliver features I hadn’t thought of, like making the digests (well, HTML version anyhow) look “prettier”.

I haven’t figured out how to copy the branch to master, so if you know please leave me a note.

Summary of changes, Digests 3.2.0 to 3.2.1

Administration Control Panel

  • Explanation text added for weekly digest day, to indicate that digests may be received on the day before the day set for users in the western hemisphere, since digests are sent based on UTC. This is already stated in the User Control Panel.
  • BEHAVIOR CHANGE: Digests time zone removed. This was used in Edit Subscribers and in Balance load to show the hour digests are sent out. The timezone for the user in the User Control Panel is now used instead. This is less confusing.
  • NEW FEATURE: Switch allows administrator to show the full forum path in digests. If set this applies to all users.
  • NEW FEATURE: Switch allows administrator to lowercase the digest type. This is at the request of a language translator.
  • BUG FIX: The default board language is no longer used as the basis for making digest language decisions in the ACP. Instead, the language you set in the UCP is used instead which is the way phpBB does it elsewhere.
    NEW FEATURE: Dates are translated into the language needed. The PHP date function only works with English formatted dates. strftime and a new function is used to translate date items like month names from English to the language desired.
  • BUG FIX: When jQuery usage was introduced into Digests, it had the effect of undoing logic written that sent to the web server only those variables that were changed on the Edit Subscriber screen on form submission. Since there are so many fields, it could cause a PHP max_input_vars error. Changed the logic so that jQuery will handle this instead of regular Javascript.
  • Cleaned up the jQuery so selectors weren’t used on multiple lines, the preferred way to do things in jQuery.
  • NEW FEATURE: Administrator can specify tags to remove from HTML digests to prevent certain security issues. Removed tags are replaced with a message indicating the tag was removed and suggesting clicking on the post link to see the full content.
  • BUG FIX: Mass subscriptions no longer set all new subscribers to the same hour. Rather if a random hour is requested, one is generated for each new subscriber. Previously after doing this you would also have to run Balance Load to distribute the digest send hour.
  • NEW FEATURE: Balance load shows the average digests sent per hour (rounded to the nearest whole number). Hours are shown in more usable format based on user’s date/time format. A list of subscribers for the hour are also now shown with their digest type annotated by the styling used (none = daily, italics = weekly, bolded = monthly). In addition, you can limit balancing to a digest type (daily, weekly or monthly), you can specify the hours you want to balance and you can specify the hours to be used for balancing (for example, to avoid sending a digest in the middle of the night).
  • User default settings now use the same language strings used in the User Control Panel interface where possible.

General

  • NEW FEATURE: Timezone hours can be prefaced with a + if the hour is positive and the sprintf markup requests it.
  • BUG FIX: Function dateFormatToStrftime will ignore AM/PM designation in date formats.
    Created many additional plural rules for language files were appropriate
  • Where there is a substitution of more than one variable in a language string, they are now marked positionally, ex: %1$d instead of %d. This aids repositioning language elements in non-English translations.
  • BUG FIX: Subscribe to digests on registration logic changed. You must be a human (i.e. omits possibility that bots could get a digest).
  • MAJOR BUG FIX: Migrations of Digests from phpBB 3.0 no longer fails because an instance of the tools factory was not first declared. (This was not necessary in phpBB 3.1.)

Template changes

  • MAJOR ENHANCEMENT: HTML digests look “prettier”. Background colors subtly change between posts within a topic making it easier to distinguish when posts begin and end. Background colors are consistent. Page background is now white. DIV boxes containing text no longer stretch across the width of the window if there is not much content.

Mailer

  • NEW FEATURE: Switch allows administrator to lowercase the first letter of the digest type.
  • NEW FEATURE: If show the forum path is enabled, the forum name in the digest will appear something like Category 1 :: Forum 1 :: Category A :: Forum B. Otherwise it will just show Forum B.
  • NEW FEATURE: If the switch that allows administrator to lowercase the digest type is enabled, behavior changes for how the board name is shown in digests. For example, in certain languages “Daily Digest” would be “Digest Daily” and will logically precede the board name. If set to yes, then the digest title will appear something like “Digest daily of my board name” depending on the translation with the first letter of the board name in lowercase.
  • BUG FIX: If for some reason the last time digests were successfully sent is after the current time, the mailer will log an error and abort. Otherwise an infinite loop starts.
  • BUG FIX: Language files are applied for each user based on their preferred language. This keeps for example a user who has their language set to French from getting English markup in the digest.
  • BUG FIX: Mailer no longer allows bots and inactive users to be selected in the initial query of users receiving digests for an hour.
  • MAJOR BUG FIX: List of posts for the time period is no longer constrained by the SQL call to limit the maximum number of digests based on the administrator or user’s maximum posts settings. This can cause other users to not see posts.
  • MAJOR BUG FIX: Fixed bug that did not correctly compute month start and end dates for monthly digests. Date function was incorrectly called generating a warning.
  • NEW FEATURE: Digest translator can now be optionally identified and linked at the bottom of the digest.
  • BUG FIX: Fixed bug that printed a “No private messages” announcement twice in text digests.
  • MAJOR BUG FIX: Fixed bug that did not properly sort posts for some digests because no array was initialized.
  • NEW FEATURE: Dates appearing in digests are now translated to show month names and days of week (including abbreviations) in the user’s language.
  • NEW FEATURE: Administrator specified tags (which may cause security problems) are removed from HTML posts. A message is substituted letting the subscriber know content was removed.

User Control Panel

  • Spacing for digest type is now consistent with other spacing for fields on the screen.

ModSecurity can cause problems with phpBB

If you are noticing weird errors on your phpBB forum which otherwise has been running well for years, ModSecurity may be causing them. I’ve been noticing a lot of issues with phpBB forums lately that have ModSecurity as the root cause.

What is ModSecurity?

It’s a web application firewall, wholly open source. As its name implies, it’s job is to tighten up the security of a web server. Web servers are prominent targets for hackers, of course. ModSecurity was originally written as a module for the Apache web server. It has lots of functions of course but its main job is to prevent hackers from damaging web servers and the websites that reside on them. Started in 2002 for Apache, it now serves all principal web servers on the web, including NGINX and IIS.

How does ModSecurity cause problems with phpBB?

It appears that ModSecurity and phpBB don’t get along very well. These problems may be occurring because the web host added it or turned it on for you and didn’t tell you. ModSecurity can generate various application errors causing the behavior of phpBB to change. In one recent example, when a client tried to click on the Administration Control Panel link, he was redirected to his website homepage instead. Disabling ModSecurity solved his problem.

Here are some other symptoms caused by ModSecurity intercepting and redirecting web traffic that my clients experienced recently:

  • In phpBB 3.2, the Viglink and share forum statistics screen come up in the Administration Control Panel by default after you update to that version. There are checkboxes that allow you to uncheck these. In this case unchecking these and submitting the form generated an unfriendly error message: “Unused” and reported an internal error. This made it impossible to get to the General tab and do things like purge the cache.
  • A forum would not come up at all. Only a white screen appeared. Disabling ModSecurity solved the problem. Note: other issues can cause this, including malware or syntax errors introduced into your forum’s phpBB code.
  • In phpBB’s /cache/production folder, files are created by phpBB with an “autoload_” prefix. These were getting deleted outside of phpBB, triggering PHP warnings. Turning off ModSecurity caused the problem to go away (after purging the cache).

How do I know if I am using ModSecurity?

Administration Control Panel > PHP Information. If your web server is Apache, search for “apache2handler” and look at the loaded modules. Scan for “mod_security” or “mod_security2”. If it’s there, it’s enabled. If you can view your web server error log, scan it for “mod_security”. If you find it, it’s enabled. You can also ask your web host if they have it enabled for your site.

Is it safe to disable ModSecurity?

Perhaps not but you may not be able to have anyone access your forum unless you disable it. Most likely ModSecurity’s rules are not optimally written to accommodate phpBB forums.

Can I disable ModSecurity just inside my forum and leave it enabled on the rest of my website?

Perhaps. Try adding this to the top or bottom your forum’s .htaccess file. If the problem goes away, you are done! (The IIS web server does not use the .htaccess file.)

<IfModule mod_security.c>
  SecFilterEngine Off
  SecFilterScanPOST Off
</IfModule>

How do I turn off ModSecurity for my domain?

Look in your web host control panel. If there is a security section, there may be a feature there to enable or disable ModSecurity. Disable it if you can find it. You may have to ask your web host.

March 2017 work summary

I was on vacation or unavailable for ten days in March, so my work in March was slimmer than normal. Here’s some of the work that I did do, along with a lot of work for teaching a Javascript class locally. All client information has been anonymized.

  • First, I helped rehost a forum running phpBB 3.0.8 to a new host. As part of rehosting I had to upgrade the site’s Wiki (MediaWiki) which took some puzzling through. I also moved a second domain into the new web space. I then upgraded the forum from phpBB 3.0.8 to 3.1.10, adding a custom style the client purchased. I added the Stop Forum Spam extension. The spambot I added for the Wiki though would not work after the Wiki was upgraded, so I installed a different Wiki extension (basically a Cleantalk extension) that would work with the new version. I moved and integrated a news database for the Wiki, changed the DNS to point to the new hosting and installed and configured a SSL certificate for the site too. Later I disabled stop forum spam extension as it wasn’t working correctly. Installed Cleantalk extension for phpBB instead. I cleared out large numbers of IP and email address bans that were preventing some users from creating accounts. 
  • I have complained about Aabaco web hosting before. For once though I was able to move a client off this miserable host, where he had an old phpBB 2 forum. phpMyAdmin was available and I discovered I could export certain tables in small chunks to get a complete export and without encountering timeouts. I was able to update the forum from phpBB 2.0.22 to phpBB 3.2.0 by moving everything to my local computer and doing the conversion there. Two issues were noted during upgrade: missed exporting the ranks table and default_lang row was missing from phpbb_config table and had to be added manually. Once I worked around issues these the conversion was able to finish. Installed the Stop Forum Spam extension. Placed the old logo on the prosilver style. Made sure “No access” permissions were set for guests and robots as the site is very private. Client asked me to install a black and white style but the result didn’t look good (probably because the version of the style was not yet approved for phpbb 3.2) so we went with prosilver. Moved database and files to Siteground hosting. Changed the nameservers to point to the new hosting. Later, the client discovered that some button images were not showing. This was because at the time the 3.2 version of the style was not official. Installing the officially approved style made the problem go away. I then installed my digests extension for phpBB 3.2.0 on the client’s site. I configured a cron job so digests would be sent hourly without fail. There was an issue because phpBB’s cron was locked, so I unlocked it via a database query. I also created a digest stylesheet since black background of the default style was an issue in rendering a readable digest.
  • I provided some advice on developing a WordPress plugin that would facilitate logging into phpBB from WordPress, basically by reading phpBB’s cookies that were normally present for the domain. User decided to write the plugin himself.
  • For an established customer, I swapped an advertising logo at top of his site after first resizing the new image.
  • A client could not upgrade his forum from phpBB 3.1.10 to 3.2.0. Doing so generated a HTTP 500 error. There were no logs with clues pointing to the problem, so I referred him to his web host. I turned out there was a syntax error in functions.php that caused the issue, which is very strange as it suggests that FTP did not move the file correctly, something so unusual I didn’t bother to look for it! The FTP protocol has error checking built in and will retransmit the file automatically if it detects a problem.
  • I upgraded a forum from phpBB 3.1.9 to 3.2.0. I installed a 3.2 version of prosilver_se style. I integrated the logo again. I added code to use his TrueType scalable fonts. I also upgraded his Tapatalk extension to the latest version and installed the latest Danish and German language packs. I added the NavBar Search extension to move search bar to the navigation bar so it did not overwrite his logo.
  • I installed my digest 3.2.0 extension on a phpBB 3.2.0 site. During installation some sort of MySQL error occurred causing host went away error messages. Lots and back and forth with tech support followed, which was of no use. Eventually I got SSH access, connected to the database via command line, was able to drop the phpbb_users table (error occurred when altering table), restored the table from client’s backup copy, then tried to install the extension again and it worked. Created a cron to make sure digests go out hourly.