Digests version 3.2.8 released

See this post on phpbb.com for details.

This version of the extension can be downloaded from my digest extension page or from the GitHub 3.2.8 branch page. When downloading from GitHub, remember that it must be copied into a blank /ext/phpbbservices/digests folder.

Of course disable the current digest extension first then remove the /ext/phpbbservices/digests/* files. Then upload the new version then enable again.

Bug in Digests extension version 3.2.7

This will hit you if a subscriber needs a table of content and you installed version 3.2.7 of the digests extension. What happened was that in cleaning up the code I decided that the assign_vars method should be simplified to use assign_var when only one template variable needs to be connected to a template. Except that the messenger class used for email is a subset of the template class and it doesn’t support the assign_var method. So it fails. You can see the patch here.

Digests 3.2.7 released

This extension is still not approved. The phpBB extension review team found more issues, all very minor.

It’s taken a long time to put out a new release because I had to wait a few months for a review, and then I had to puzzle my way through bugs that were introduced into phpBB’s migrator in phpBB 3.2.2 which in many cases kept digests from installing correctly. It’s all pretty annoying but issues like this are bound to creep in from time to time. Anyhow, version 3.2.7 is released. With this release I’ve retested installing digests from a database upgraded from phpBB 3.0 with the digests mod installed. Apparently I hadn’t retested this since the 3.1 version of digests. Using this approach you can retain your digest settings from the mod.

You can download this version from my digests extension page or on the GitHub branch.

Most of the change log is pretty technical and arguably irrelevant but you should pay attention to bug fixes and functionality changes.

Bug fixes:

  • Fixed HTML5 validation issue with Edit Subscribers, closing </textarea> was missing
  • Fixed HTML5 validation issue with Balance Load, added three <col> tags
  • UCP dialog error fixed by adding additional code to jQueryUI interface
  • Moved posts are not included in a digest
  • Configuration variable phpbbservices_digests_strip_tags added by the migrator, was missed from earlier version

Changes in functionality:

  • When the manual mailer is run, the file name includes a hash for added security

Extension team review fixes:

  • Replaced isset($_REQUEST[‘config’]) with $this->request->is_set(‘config’)
  • Replaced addslashes($balance) with $this->db->sql_escape($balance)
  • Removed !defined(‘IN_PHPBB’) in mailer
  • Moved code to load messenger class inside the mail_digests function in cron/task/digests.php

Best practices:

  • Replaced sizeof() with count() as sizeof() is an alias of count()
  • Removed redundant conditional operators where unnecessary
  • Removed unneeded casting of function return values where unnecessary
  • Used $html_messenger->assign_var() when only one template variable needed to be set instead of $html_messenger->assign_vars()

CSS:

  • Included ACP and UCP .css file through a switch, so they would not be loaded by other modules
  • Edit subscribers now shows alternating row colors properly

Javascript:

  • Included ACP and UCP .js file through a switch, so they would not be loaded by other modules
  • Fixed some variable names that were not using camelcase
  • addToStack() function logic now uses jQuery
  • Removed redundant conditional operators where unnecessary
  • Added var in front of variables that did not use them

ACP Interface:

  • Removed query for f_list privilege, unneeded
  • Used constants DIGESTS_DAILY_VALUE, DIGESTS_WEEKLY_VALUE and DIGESTS_MONTHLY_VALUE instead of literals
  • Properly indented code in templates

UCP Interface:

  • Properly indented code in templates

services.xml:

  • helper class is no longer passed to main_listener class as it is no longer needed

html_messenger.php:

  • Brought copied functions up to date with functions in version 3.2.2 of /includes/functions_messenger.php

Migrations:

  • Lots of work to simplify and address issues, some related to bugs in the migrator introduced in phpBB 3.2.2
  • Conversion from digests mod: now properly converts configuration variables, modules and schema changes during enabling the extension if these remain in the database after upgrading from phpBB 3.0. This allows digest settings to be retained. Apparently this functionality never worked on phpBB 3.2. To get around what look like migrator bugs in phpBB 3.2.2, old modules are deleted with a direct database call. Note: if AutoMOD was used to install Digests, you need to remove AutoMOD modules manually. The symptom is an ACT_CAT_MOD tab appears in the ACP.
  • Updates configuration variable to point to new digests web page URL
  • Helper container removed as it is no longer needed
  • release_3_0_6.php removed
  • common.php linkage removed as it is not used anymore

Issues installing the digests extension on phpBB 3.2 when digest mod was installed on phpBB 3.0

For reasons that are not entirely clear to me there are issues when installing the digests extension on phpBB 3.2 if you are carrying over digest settings that were in place on phpBB 3.0 where the digest modification was installed. Most likely there were subtle changes in the phpBB modules library. Please see the first post of the digests extension topic for current instructions to use in this case.  The new additions are copied below:

You should remove any old AutoMOD and digests modules before installing Digests.

Since the digests mod was often installed using AutoMOD, there may be a bogus ACP_CAT_MODS tab when you go into the Administration Control Panel. phpBB’s upgrade program isn’t smart enough to remove these. Clicking on the tab will usually trigger an error that tells you the module for AutoMOD no longer exists. (Of course it should not. The AutoMOD program should have been removed with the upgrade of your forum from 3.0 to 3.1 or 3.2.) These can be manually removed on the System Tab, Module Management, Administration Control Panel. You will have to descend the module tree and remove modules from the bottom up, eventually removing the tab altogether. You must do this prior to installing digests if you are importing digest settings from what was at one time a phpBB 3.0 database with the digest mod installed. If you don’t, you will likely get errors when trying to enable the digests extension. Make sure you remove all modules under this tab and what used to be the .MOD tab (now ACP_CAT_MODS tab).

It’s recommended that you remove old digest modules through the database before installing digests

The installer was designed to clean these up. However from what looks like quirks in the phpBB 3.2 modules library, you may get errors. Enabling the extension again usually solves the problem. It’s better to be proactive and use a program like phpMyAdmin to remove these from the database instead:

  • Check your config.php file to make sure you have the correct table prefix. In the example below, phpbb_ is assumed for the table prefix.
  • Open a database manipulation program, typically phpMyAdmin
  • Make sure you have selected the correct database. The config.php file will indicate the database name.
  • If you are using phpMyAdmin, select the SQL tab
  • Execute the following SQL, changing the table prefix if necessary: DELETE FROM phpbb_modules WHERE module_langname LIKE '%DIGEST%'
  • In phpBB, purge the cache in the Administration Control Panel

Digests 3.2.6 (not RC13 this time) released

Please see this post for details. You can download the latest version from my digests page or get it here in GitHub.

There’s really no new functionality but it does clean up a few bugs, changes in minor ways the look of some ACP pages (Edit subscribers mostly) and addresses the issues by the last review by the phpBB extensions team. It’s been submitted for review by the extensions team and most likely will get approved.

It took about 6 weeks to get a review last time, so I expect it will take about that long this time too. But maybe since it is reaching a release stage they will expedite the process.

Digests 3.2.6 (RC13) released

This version incorporates the bug that the Czech translator for digests, Petr Hendl found and fixed. You can download it from GitHub or on my digests page.

In my testing I was only able to get digests to mail from my web server when the email package size was 0. I assumed this was something about how my web server email interface was configured, which I can’t control.

I also assumed that when $html_messenger->send() was called it placed it into the queue unless email_package_size = 0. It’s counter-intuitive to call $html_messenger->send() and then have to call $html_messenger->save_queue() afterward to ensure it gets in the queue. Thanks to Petr, he determined this was the way it is apparently done.

So if you are using digests, after you install this release feel free to change your email package size from 0.

Digests 3.2.5 (RC12) released

Digests version 3.2.5-RC12 is released! In addition, it’s been submitted for extension team review. With luck it may finally get approved. Given its complexity I expect they will reject it again. I expect it will take a few months to get a review, based on past experience.

You can download the archive from my digests page.

Not much new, fixed some bugs, most changes are to the ACP:

  • Templates were converted to use TWIG syntax. Minor HTML5 validation errors were also fixed.
  • Fixed bug in Edit Subscribers that would store a user_digest_send_hour_gmt outside the valid values of 0 – 23 in certain cases
  • Balance load now has individual columns to show daily, weekly and monthly subscribers for a particular hour rather than use bold and italics to distinguish digest types
  • Fixed bug in get_subscribers_for_hour function. The second parameter for the function is supposed to be an offset in hours from UTC.
  • ACP edit subscribers and balance load functions now have different background colors for alternating table rows
  • As many inline styles as possible were moved into stylesheets
  • Fixed various SQL statements to cast values into integers or floats explicitly. This is a phpBB coding standards requirement.
  • In the admin log, dates now actually show two digit months and day
  • includes/html_messenger.php, which overrides the messenger class, was updated to include new code added in /includes/functions_messenger.php since the html_messenger class was first created

Digests 3.2.4 (RC11) released

Finally, a new release of digests! What was the hold up? Busy doing other things, vacations, but also it’s very hard to test all the permutations and to fix bugs that manifest only in cron modes. Such bugs must be tracked down by sending messages to the Admin log through debug statements. Very tedious!

As for what’s new, really except for closing bugs there is no new functionality. It should be more stable and reliable. You can download the archive on my Digests page or on GitHub. If upgrading, make sure to follow standard procedures. The first post of the digests topic provides a lot of guidance.

Change log:

New features

  • None

Major bugs fixed

  • Fixed bug in mailer that incorrectly reported the hour that was run using 12 hour format (0-12) instead of 24 hour format (0-24), leading to false log entries such as hour 1 could actually have been hour 13
  • Fixed bad positional parameters in the DIGESTS_DISCLAIMER language variable, fix from petr-hendl pull request (thanks)
  • Fixed bug that creates a user timezone object if it does not exist. In cron mode sometimes it was not present or deallocated. It is needed to make dates and hours print out in a language independent fashion.

Minor bugs fixed

  • All hours and time zone offsets are cast as floats since time zones are not necessarily offset by an integer from UTC.
  • Default text for maximum display words in the ACP is no longer the same as what is in the UCP. Needed to revert to the previous language string because -1 in this value will show full display text by default.
  • Language string LOG_CONFIG_DIGESTS_HOUR_RUN uses positional parameters, fix from petr-hendl pull request (thanks)
  • When digests are run manually they are no longer stopped if it’s been less than an hour since a digests was sent to the same user
  • Fixed certain admin log entries that were added even if the logging option was turned off

Tweaks and improvements

  • New check_send_hour helper function reduces duplicative code and centralizes all send hour offset logic
  • PHP and template variables with gmt in the name are changed to utc
  • <br /> changed to <br> when used inside of variables and expressions
  • <hr /> tags in templates changed to <hr> when used inside of variables and expressions
  • Constants are used to identify the mode the mailer is running (manual, regular and phpbb), makes code easier to read
  • Function validateDate renamed validate_date for consistency
  • Pretty quotes &rsquo;, &ldquo; and &rdquo; in language strings since this was an issue flagged by the extension review team while reviewing my Smartfeed extension
  • Removed language string LOG_CONFIG_DIGESTS_DIRECTORY_CREATE_ERROR (no longer needed because this error is not easily trapped in ext.php)
  • digests_html.txt uses new .newline class which makes it less likely digests will look odd, from petr-hendl pull request (thanks)
  • Version check checks a new file specific to phpBB 3.2 releases
  • Logic for creating and removing a /cache/phpbbservices/digests folder moved into ext.php from /cron/task/digests.php where it logically belongs as it should happen when the extension is installed
  • Constructor for digests class is fully documented
  • Moved board disabled check (which keeps digests from going out when the board is disabled) into the should_run function of the cron
  • Made the logic in the run function of the mailer less complicated
  • Since largely the same code was used to create attachment markup for both posts and private messages, moved it into a function
  • Log date format shows two digit months, days and hours

 

Digests “error while creating image” fix

Many digest users have been having issues running digests, particularly versions 3.2.2 and 3.2.3. At the suggestion of a user I introduced a feature that allows month abbreviations and days of week to be language independent by using phpBB’s $user->format_date() function. Aside from most if not all of digests not going out for a particular hour, while you will see a “Starting digests mailer” entry in the admin log you won’t see an “Ending digests mailer”. In the phpBB error log you should see this:

Error while creating image
» Error in [ROOT]/phpbb/user.php on line 599: DateTime::setTimezone() expects parameter 1 to be DateTimeZone, null given

What is going on is that phpBB expects a PHP DateTimeZone object to be created and attached to the important $user object as $user->timezone. If it’s not there, the error will be triggered because the object doesn’t exist:

 $time->setTimezone($this->timezone);

In my testing, this happens when “Anonymous” (the guest account, or user_id = 1) is running digests. Essentially “Anonymous” is hitting the forum and triggering phpBB’s cron. phpBB’s cron may trigger the digest cron (if an hour has elapsed since digests were last run). Even though “Anonymous” has a timezone associated with the account (GMT) either the timezone object is not getting created or it is getting destroyed when it goes through the loop that sends out all digests for a given hour.

The solution is documented here and should be applied to versions 3.2.3 and 3.2.2 if that is installed. The instructions assume 3.2.3 is installed. Essentially before calling $user->format_date() it tests to see if the timezone object exists and if it doesn’t it creates it. There are three possible instances that could trigger this, so all should be fixed.

There is a general problem with the digests architecture. It extensively uses phpBB’s built in tools, such as this library, but also the templating system to create pretty digests, which assumes a live user is interacting with the forum. However, digests are often run by “Anonymous”, which is not quite a “full” user, or by a system cron. phpBB was clearly not designed to use these features (at least not very well) in “cron” mode. So as these problems are discovered, they must be found and fixed. And unfortunately these sorts of problems are devilishly hard to track down as they can’t be discovered interactively and only through tedious debugging of a sufficiently large forum by adding entries to the log. One of my clients essentially used his large forum to help me troubleshoot the issue and fix it.

The problem is not manifest when manually running the mailer. I can’t actually test digests from the command line (I need an email server in my development environment to do that) so I have to test in phpBB cron mode. A sufficiently large test database might have revealed this problem, but of course digests are highly dynamic so it’s hard to set up one of these.

Changes will get published to GitHub and this will probably be part of a new version, but not right away. I started reviewing the code to clean it up, and need to integrate these changes into that (and test it!) before releasing the next version.