Digests 3.2.7 released

This extension is still not approved. The phpBB extension review team found more issues, all very minor.

It’s taken a long time to put out a new release because I had to wait a few months for a review, and then I had to puzzle my way through bugs that were introduced into phpBB’s migrator in phpBB 3.2.2 which in many cases kept digests from installing correctly. It’s all pretty annoying but issues like this are bound to creep in from time to time. Anyhow, version 3.2.7 is released. With this release I’ve retested installing digests from a database upgraded from phpBB 3.0 with the digests mod installed. Apparently I hadn’t retested this since the 3.1 version of digests. Using this approach you can retain your digest settings from the mod.

You can download this version from my digests extension page or on the GitHub branch.

Most of the change log is pretty technical and arguably irrelevant but you should pay attention to bug fixes and functionality changes.

Bug fixes:

  • Fixed HTML5 validation issue with Edit Subscribers, closing </textarea> was missing
  • Fixed HTML5 validation issue with Balance Load, added three <col> tags
  • UCP dialog error fixed by adding additional code to jQueryUI interface
  • Moved posts are not included in a digest
  • Configuration variable phpbbservices_digests_strip_tags added by the migrator, was missed from earlier version

Changes in functionality:

  • When the manual mailer is run, the file name includes a hash for added security

Extension team review fixes:

  • Replaced isset($_REQUEST[‘config’]) with $this->request->is_set(‘config’)
  • Replaced addslashes($balance) with $this->db->sql_escape($balance)
  • Removed !defined(‘IN_PHPBB’) in mailer
  • Moved code to load messenger class inside the mail_digests function in cron/task/digests.php

Best practices:

  • Replaced sizeof() with count() as sizeof() is an alias of count()
  • Removed redundant conditional operators where unnecessary
  • Removed unneeded casting of function return values where unnecessary
  • Used $html_messenger->assign_var() when only one template variable needed to be set instead of $html_messenger->assign_vars()

CSS:

  • Included ACP and UCP .css file through a switch, so they would not be loaded by other modules
  • Edit subscribers now shows alternating row colors properly

Javascript:

  • Included ACP and UCP .js file through a switch, so they would not be loaded by other modules
  • Fixed some variable names that were not using camelcase
  • addToStack() function logic now uses jQuery
  • Removed redundant conditional operators where unnecessary
  • Added var in front of variables that did not use them

ACP Interface:

  • Removed query for f_list privilege, unneeded
  • Used constants DIGESTS_DAILY_VALUE, DIGESTS_WEEKLY_VALUE and DIGESTS_MONTHLY_VALUE instead of literals
  • Properly indented code in templates

UCP Interface:

  • Properly indented code in templates

services.xml:

  • helper class is no longer passed to main_listener class as it is no longer needed

html_messenger.php:

  • Brought copied functions up to date with functions in version 3.2.2 of /includes/functions_messenger.php

Migrations:

  • Lots of work to simplify and address issues, some related to bugs in the migrator introduced in phpBB 3.2.2
  • Conversion from digests mod: now properly converts configuration variables, modules and schema changes during enabling the extension if these remain in the database after upgrading from phpBB 3.0. This allows digest settings to be retained. Apparently this functionality never worked on phpBB 3.2. To get around what look like migrator bugs in phpBB 3.2.2, old modules are deleted with a direct database call. Note: if AutoMOD was used to install Digests, you need to remove AutoMOD modules manually. The symptom is an ACT_CAT_MOD tab appears in the ACP.
  • Updates configuration variable to point to new digests web page URL
  • Helper container removed as it is no longer needed
  • release_3_0_6.php removed
  • common.php linkage removed as it is not used anymore

Issues installing the digests extension on phpBB 3.2 when digest mod was installed on phpBB 3.0

For reasons that are not entirely clear to me there are issues when installing the digests extension on phpBB 3.2 if you are carrying over digest settings that were in place on phpBB 3.0 where the digest modification was installed. Most likely there were subtle changes in the phpBB modules library. Please see the first post of the digests extension topic for current instructions to use in this case.  The new additions are copied below:

You should remove any old AutoMOD and digests modules before installing Digests.

Since the digests mod was often installed using AutoMOD, there may be a bogus ACP_CAT_MODS tab when you go into the Administration Control Panel. phpBB’s upgrade program isn’t smart enough to remove these. Clicking on the tab will usually trigger an error that tells you the module for AutoMOD no longer exists. (Of course it should not. The AutoMOD program should have been removed with the upgrade of your forum from 3.0 to 3.1 or 3.2.) These can be manually removed on the System Tab, Module Management, Administration Control Panel. You will have to descend the module tree and remove modules from the bottom up, eventually removing the tab altogether. You must do this prior to installing digests if you are importing digest settings from what was at one time a phpBB 3.0 database with the digest mod installed. If you don’t, you will likely get errors when trying to enable the digests extension. Make sure you remove all modules under this tab and what used to be the .MOD tab (now ACP_CAT_MODS tab).

It’s recommended that you remove old digest modules through the database before installing digests

The installer was designed to clean these up. However from what looks like quirks in the phpBB 3.2 modules library, you may get errors. Enabling the extension again usually solves the problem. It’s better to be proactive and use a program like phpMyAdmin to remove these from the database instead:

  • Check your config.php file to make sure you have the correct table prefix. In the example below, phpbb_ is assumed for the table prefix.
  • Open a database manipulation program, typically phpMyAdmin
  • Make sure you have selected the correct database. The config.php file will indicate the database name.
  • If you are using phpMyAdmin, select the SQL tab
  • Execute the following SQL, changing the table prefix if necessary: DELETE FROM phpbb_modules WHERE module_langname LIKE '%DIGEST%'
  • In phpBB, purge the cache in the Administration Control Panel

Digests 3.2.6 (not RC13 this time) released

Please see this post for details. You can download the latest version from my digests page or get it here in GitHub.

There’s really no new functionality but it does clean up a few bugs, changes in minor ways the look of some ACP pages (Edit subscribers mostly) and addresses the issues by the last review by the phpBB extensions team. It’s been submitted for review by the extensions team and most likely will get approved.

It took about 6 weeks to get a review last time, so I expect it will take about that long this time too. But maybe since it is reaching a release stage they will expedite the process.

Digests 3.2.6 (RC13) released

This version incorporates the bug that the Czech translator for digests, Petr Hendl found and fixed. You can download it from GitHub or on my digests page.

In my testing I was only able to get digests to mail from my web server when the email package size was 0. I assumed this was something about how my web server email interface was configured, which I can’t control.

I also assumed that when $html_messenger->send() was called it placed it into the queue unless email_package_size = 0. It’s counter-intuitive to call $html_messenger->send() and then have to call $html_messenger->save_queue() afterward to ensure it gets in the queue. Thanks to Petr, he determined this was the way it is apparently done.

So if you are using digests, after you install this release feel free to change your email package size from 0.

Digests 3.2.5 (RC12) released

Digests version 3.2.5-RC12 is released! In addition, it’s been submitted for extension team review. With luck it may finally get approved. Given its complexity I expect they will reject it again. I expect it will take a few months to get a review, based on past experience.

You can download the archive from my digests page.

Not much new, fixed some bugs, most changes are to the ACP:

  • Templates were converted to use TWIG syntax. Minor HTML5 validation errors were also fixed.
  • Fixed bug in Edit Subscribers that would store a user_digest_send_hour_gmt outside the valid values of 0 – 23 in certain cases
  • Balance load now has individual columns to show daily, weekly and monthly subscribers for a particular hour rather than use bold and italics to distinguish digest types
  • Fixed bug in get_subscribers_for_hour function. The second parameter for the function is supposed to be an offset in hours from UTC.
  • ACP edit subscribers and balance load functions now have different background colors for alternating table rows
  • As many inline styles as possible were moved into stylesheets
  • Fixed various SQL statements to cast values into integers or floats explicitly. This is a phpBB coding standards requirement.
  • In the admin log, dates now actually show two digit months and day
  • includes/html_messenger.php, which overrides the messenger class, was updated to include new code added in /includes/functions_messenger.php since the html_messenger class was first created

Digests 3.2.4 (RC11) released

Finally, a new release of digests! What was the hold up? Busy doing other things, vacations, but also it’s very hard to test all the permutations and to fix bugs that manifest only in cron modes. Such bugs must be tracked down by sending messages to the Admin log through debug statements. Very tedious!

As for what’s new, really except for closing bugs there is no new functionality. It should be more stable and reliable. You can download the archive on my Digests page or on GitHub. If upgrading, make sure to follow standard procedures. The first post of the digests topic provides a lot of guidance.

Change log:

New features

  • None

Major bugs fixed

  • Fixed bug in mailer that incorrectly reported the hour that was run using 12 hour format (0-12) instead of 24 hour format (0-24), leading to false log entries such as hour 1 could actually have been hour 13
  • Fixed bad positional parameters in the DIGESTS_DISCLAIMER language variable, fix from petr-hendl pull request (thanks)
  • Fixed bug that creates a user timezone object if it does not exist. In cron mode sometimes it was not present or deallocated. It is needed to make dates and hours print out in a language independent fashion.

Minor bugs fixed

  • All hours and time zone offsets are cast as floats since time zones are not necessarily offset by an integer from UTC.
  • Default text for maximum display words in the ACP is no longer the same as what is in the UCP. Needed to revert to the previous language string because -1 in this value will show full display text by default.
  • Language string LOG_CONFIG_DIGESTS_HOUR_RUN uses positional parameters, fix from petr-hendl pull request (thanks)
  • When digests are run manually they are no longer stopped if it’s been less than an hour since a digests was sent to the same user
  • Fixed certain admin log entries that were added even if the logging option was turned off

Tweaks and improvements

  • New check_send_hour helper function reduces duplicative code and centralizes all send hour offset logic
  • PHP and template variables with gmt in the name are changed to utc
  • <br /> changed to <br> when used inside of variables and expressions
  • <hr /> tags in templates changed to <hr> when used inside of variables and expressions
  • Constants are used to identify the mode the mailer is running (manual, regular and phpbb), makes code easier to read
  • Function validateDate renamed validate_date for consistency
  • Pretty quotes &rsquo;, &ldquo; and &rdquo; in language strings since this was an issue flagged by the extension review team while reviewing my Smartfeed extension
  • Removed language string LOG_CONFIG_DIGESTS_DIRECTORY_CREATE_ERROR (no longer needed because this error is not easily trapped in ext.php)
  • digests_html.txt uses new .newline class which makes it less likely digests will look odd, from petr-hendl pull request (thanks)
  • Version check checks a new file specific to phpBB 3.2 releases
  • Logic for creating and removing a /cache/phpbbservices/digests folder moved into ext.php from /cron/task/digests.php where it logically belongs as it should happen when the extension is installed
  • Constructor for digests class is fully documented
  • Moved board disabled check (which keeps digests from going out when the board is disabled) into the should_run function of the cron
  • Made the logic in the run function of the mailer less complicated
  • Since largely the same code was used to create attachment markup for both posts and private messages, moved it into a function
  • Log date format shows two digit months, days and hours

 

Digests “error while creating image” fix

Many digest users have been having issues running digests, particularly versions 3.2.2 and 3.2.3. At the suggestion of a user I introduced a feature that allows month abbreviations and days of week to be language independent by using phpBB’s $user->format_date() function. Aside from most if not all of digests not going out for a particular hour, while you will see a “Starting digests mailer” entry in the admin log you won’t see an “Ending digests mailer”. In the phpBB error log you should see this:

Error while creating image
» Error in [ROOT]/phpbb/user.php on line 599: DateTime::setTimezone() expects parameter 1 to be DateTimeZone, null given

What is going on is that phpBB expects a PHP DateTimeZone object to be created and attached to the important $user object as $user->timezone. If it’s not there, the error will be triggered because the object doesn’t exist:

 $time->setTimezone($this->timezone);

In my testing, this happens when “Anonymous” (the guest account, or user_id = 1) is running digests. Essentially “Anonymous” is hitting the forum and triggering phpBB’s cron. phpBB’s cron may trigger the digest cron (if an hour has elapsed since digests were last run). Even though “Anonymous” has a timezone associated with the account (GMT) either the timezone object is not getting created or it is getting destroyed when it goes through the loop that sends out all digests for a given hour.

The solution is documented here and should be applied to versions 3.2.3 and 3.2.2 if that is installed. The instructions assume 3.2.3 is installed. Essentially before calling $user->format_date() it tests to see if the timezone object exists and if it doesn’t it creates it. There are three possible instances that could trigger this, so all should be fixed.

There is a general problem with the digests architecture. It extensively uses phpBB’s built in tools, such as this library, but also the templating system to create pretty digests, which assumes a live user is interacting with the forum. However, digests are often run by “Anonymous”, which is not quite a “full” user, or by a system cron. phpBB was clearly not designed to use these features (at least not very well) in “cron” mode. So as these problems are discovered, they must be found and fixed. And unfortunately these sorts of problems are devilishly hard to track down as they can’t be discovered interactively and only through tedious debugging of a sufficiently large forum by adding entries to the log. One of my clients essentially used his large forum to help me troubleshoot the issue and fix it.

The problem is not manifest when manually running the mailer. I can’t actually test digests from the command line (I need an email server in my development environment to do that) so I have to test in phpBB cron mode. A sufficiently large test database might have revealed this problem, but of course digests are highly dynamic so it’s hard to set up one of these.

Changes will get published to GitHub and this will probably be part of a new version, but not right away. I started reviewing the code to clean it up, and need to integrate these changes into that (and test it!) before releasing the next version.

Digests 3.2.3 (RC10) Released

In this version (compared with 3.2.2):

  • Improvement: $user->format_date() is used to create dates that show correct date abbreviations based on the language pack used. Thanks to Petr Hendl, the Czech digest extension translator, for finding this built-in approach. Consequently, I removed the dateFormatToStrftime() function.
  • Bug fix: Balance load previously ignored inactive users only. Now it also ignores bots, meaning normal users and founders are the only user types balanced.
  • <br /> tags changed to <br> tags, since HTML 5 is not particular like XHTML.
  • Improvement: make_tz_offset() checks for invalid user_timezone strings. If it is invalid, UTC is used for the timezone.
  • Bug fix: Fixed a bug in the validateDate() function. It was incorrectly finding every user_timezone string invalid.
  • Improvement: The digests cron calculates the date and hour digests were last run successfully, and uses this to determine if digests should be run. This should solve the potential hour creep problem and allow digests to go out for the hour on the hour even if the last time they were successfully sent out was not on the hour.
  • Bug fix: The cron includes includes/functions_content.php only if it is not already loaded.
  • Bug fix: Created a circuit breaker that prohibits a digest from going out as a duplicate if the timestamp for the time a user last received a digest indicates it is for the current date and hour.
  • The cron adds a language pack only if it has not already been loaded.
  • Improvement: Curly single and double quotes are now used in language strings.
  • All hours processed by the cron show successful completion, even though individual digests may not go out successfully. This keeps the cron from getting stuck indefinitely at a certain date and hour.

This version can be downloaded from GitHub or you can grab it off the Digests page.

Digests 3.2.2-RC9 Released

I noticed an obscure but critical bug in Edit Subscribers that misreported the hour the digest was sent in some cases, so a new release was warranted so quickly after the last one. In addition, the display hour in Edit Subscribers show shows a formatted hour. For example, it used to show 14, now it will show 2 PM if user_dateformat requests hours to show as AM or PM.

That’s pretty much it, except for updating the version number and giving some better names to variables.

To get the latest version, go to the digests page.