Digests 3.2.6 (not RC13 this time) released

Please see this post for details. You can download the latest version from my digests page or get it here in GitHub.

There’s really no new functionality but it does clean up a few bugs, changes in minor ways the look of some ACP pages (Edit subscribers mostly) and addresses the issues by the last review by the phpBB extensions team. It’s been submitted for review by the extensions team and most likely will get approved.

It took about 6 weeks to get a review last time, so I expect it will take about that long this time too. But maybe since it is reaching a release stage they will expedite the process.

Digests 3.2.6 (RC13) released

This version incorporates the bug that the Czech translator for digests, Petr Hendl found and fixed. You can download it from GitHub or on my digests page.

In my testing I was only able to get digests to mail from my web server when the email package size was 0. I assumed this was something about how my web server email interface was configured, which I can’t control.

I also assumed that when $html_messenger->send() was called it placed it into the queue unless email_package_size = 0. It’s counter-intuitive to call $html_messenger->send() and then have to call $html_messenger->save_queue() afterward to ensure it gets in the queue. Thanks to Petr, he determined this was the way it is apparently done.

So if you are using digests, after you install this release feel free to change your email package size from 0.

Digests 3.2.5 (RC12) released

Digests version 3.2.5-RC12 is released! In addition, it’s been submitted for extension team review. With luck it may finally get approved. Given its complexity I expect they will reject it again. I expect it will take a few months to get a review, based on past experience.

You can download the archive from my digests page.

Not much new, fixed some bugs, most changes are to the ACP:

  • Templates were converted to use TWIG syntax. Minor HTML5 validation errors were also fixed.
  • Fixed bug in Edit Subscribers that would store a user_digest_send_hour_gmt outside the valid values of 0 – 23 in certain cases
  • Balance load now has individual columns to show daily, weekly and monthly subscribers for a particular hour rather than use bold and italics to distinguish digest types
  • Fixed bug in get_subscribers_for_hour function. The second parameter for the function is supposed to be an offset in hours from UTC.
  • ACP edit subscribers and balance load functions now have different background colors for alternating table rows
  • As many inline styles as possible were moved into stylesheets
  • Fixed various SQL statements to cast values into integers or floats explicitly. This is a phpBB coding standards requirement.
  • In the admin log, dates now actually show two digit months and day
  • includes/html_messenger.php, which overrides the messenger class, was updated to include new code added in /includes/functions_messenger.php since the html_messenger class was first created

Digests 3.2.4 (RC11) released

Finally, a new release of digests! What was the hold up? Busy doing other things, vacations, but also it’s very hard to test all the permutations and to fix bugs that manifest only in cron modes. Such bugs must be tracked down by sending messages to the Admin log through debug statements. Very tedious!

As for what’s new, really except for closing bugs there is no new functionality. It should be more stable and reliable. You can download the archive on my Digests page or on GitHub. If upgrading, make sure to follow standard procedures. The first post of the digests topic provides a lot of guidance.

Change log:

New features

  • None

Major bugs fixed

  • Fixed bug in mailer that incorrectly reported the hour that was run using 12 hour format (0-12) instead of 24 hour format (0-24), leading to false log entries such as hour 1 could actually have been hour 13
  • Fixed bad positional parameters in the DIGESTS_DISCLAIMER language variable, fix from petr-hendl pull request (thanks)
  • Fixed bug that creates a user timezone object if it does not exist. In cron mode sometimes it was not present or deallocated. It is needed to make dates and hours print out in a language independent fashion.

Minor bugs fixed

  • All hours and time zone offsets are cast as floats since time zones are not necessarily offset by an integer from UTC.
  • Default text for maximum display words in the ACP is no longer the same as what is in the UCP. Needed to revert to the previous language string because -1 in this value will show full display text by default.
  • Language string LOG_CONFIG_DIGESTS_HOUR_RUN uses positional parameters, fix from petr-hendl pull request (thanks)
  • When digests are run manually they are no longer stopped if it’s been less than an hour since a digests was sent to the same user
  • Fixed certain admin log entries that were added even if the logging option was turned off

Tweaks and improvements

  • New check_send_hour helper function reduces duplicative code and centralizes all send hour offset logic
  • PHP and template variables with gmt in the name are changed to utc
  • <br /> changed to <br> when used inside of variables and expressions
  • <hr /> tags in templates changed to <hr> when used inside of variables and expressions
  • Constants are used to identify the mode the mailer is running (manual, regular and phpbb), makes code easier to read
  • Function validateDate renamed validate_date for consistency
  • Pretty quotes &rsquo;, &ldquo; and &rdquo; in language strings since this was an issue flagged by the extension review team while reviewing my Smartfeed extension
  • Removed language string LOG_CONFIG_DIGESTS_DIRECTORY_CREATE_ERROR (no longer needed because this error is not easily trapped in ext.php)
  • digests_html.txt uses new .newline class which makes it less likely digests will look odd, from petr-hendl pull request (thanks)
  • Version check checks a new file specific to phpBB 3.2 releases
  • Logic for creating and removing a /cache/phpbbservices/digests folder moved into ext.php from /cron/task/digests.php where it logically belongs as it should happen when the extension is installed
  • Constructor for digests class is fully documented
  • Moved board disabled check (which keeps digests from going out when the board is disabled) into the should_run function of the cron
  • Made the logic in the run function of the mailer less complicated
  • Since largely the same code was used to create attachment markup for both posts and private messages, moved it into a function
  • Log date format shows two digit months, days and hours

 

Digests “error while creating image” fix

Many digest users have been having issues running digests, particularly versions 3.2.2 and 3.2.3. At the suggestion of a user I introduced a feature that allows month abbreviations and days of week to be language independent by using phpBB’s $user->format_date() function. Aside from most if not all of digests not going out for a particular hour, while you will see a “Starting digests mailer” entry in the admin log you won’t see an “Ending digests mailer”. In the phpBB error log you should see this:

Error while creating image
» Error in [ROOT]/phpbb/user.php on line 599: DateTime::setTimezone() expects parameter 1 to be DateTimeZone, null given

What is going on is that phpBB expects a PHP DateTimeZone object to be created and attached to the important $user object as $user->timezone. If it’s not there, the error will be triggered because the object doesn’t exist:

 $time->setTimezone($this->timezone);

In my testing, this happens when “Anonymous” (the guest account, or user_id = 1) is running digests. Essentially “Anonymous” is hitting the forum and triggering phpBB’s cron. phpBB’s cron may trigger the digest cron (if an hour has elapsed since digests were last run). Even though “Anonymous” has a timezone associated with the account (GMT) either the timezone object is not getting created or it is getting destroyed when it goes through the loop that sends out all digests for a given hour.

The solution is documented here and should be applied to versions 3.2.3 and 3.2.2 if that is installed. The instructions assume 3.2.3 is installed. Essentially before calling $user->format_date() it tests to see if the timezone object exists and if it doesn’t it creates it. There are three possible instances that could trigger this, so all should be fixed.

There is a general problem with the digests architecture. It extensively uses phpBB’s built in tools, such as this library, but also the templating system to create pretty digests, which assumes a live user is interacting with the forum. However, digests are often run by “Anonymous”, which is not quite a “full” user, or by a system cron. phpBB was clearly not designed to use these features (at least not very well) in “cron” mode. So as these problems are discovered, they must be found and fixed. And unfortunately these sorts of problems are devilishly hard to track down as they can’t be discovered interactively and only through tedious debugging of a sufficiently large forum by adding entries to the log. One of my clients essentially used his large forum to help me troubleshoot the issue and fix it.

The problem is not manifest when manually running the mailer. I can’t actually test digests from the command line (I need an email server in my development environment to do that) so I have to test in phpBB cron mode. A sufficiently large test database might have revealed this problem, but of course digests are highly dynamic so it’s hard to set up one of these.

Changes will get published to GitHub and this will probably be part of a new version, but not right away. I started reviewing the code to clean it up, and need to integrate these changes into that (and test it!) before releasing the next version.

Digests 3.2.3 (RC10) Released

In this version (compared with 3.2.2):

  • Improvement: $user->format_date() is used to create dates that show correct date abbreviations based on the language pack used. Thanks to Petr Hendl, the Czech digest extension translator, for finding this built-in approach. Consequently, I removed the dateFormatToStrftime() function.
  • Bug fix: Balance load previously ignored inactive users only. Now it also ignores bots, meaning normal users and founders are the only user types balanced.
  • <br /> tags changed to <br> tags, since HTML 5 is not particular like XHTML.
  • Improvement: make_tz_offset() checks for invalid user_timezone strings. If it is invalid, UTC is used for the timezone.
  • Bug fix: Fixed a bug in the validateDate() function. It was incorrectly finding every user_timezone string invalid.
  • Improvement: The digests cron calculates the date and hour digests were last run successfully, and uses this to determine if digests should be run. This should solve the potential hour creep problem and allow digests to go out for the hour on the hour even if the last time they were successfully sent out was not on the hour.
  • Bug fix: The cron includes includes/functions_content.php only if it is not already loaded.
  • Bug fix: Created a circuit breaker that prohibits a digest from going out as a duplicate if the timestamp for the time a user last received a digest indicates it is for the current date and hour.
  • The cron adds a language pack only if it has not already been loaded.
  • Improvement: Curly single and double quotes are now used in language strings.
  • All hours processed by the cron show successful completion, even though individual digests may not go out successfully. This keeps the cron from getting stuck indefinitely at a certain date and hour.

This version can be downloaded from GitHub or you can grab it off the Digests page.

Digests 3.2.2-RC9 Released

I noticed an obscure but critical bug in Edit Subscribers that misreported the hour the digest was sent in some cases, so a new release was warranted so quickly after the last one. In addition, the display hour in Edit Subscribers show shows a formatted hour. For example, it used to show 14, now it will show 2 PM if user_dateformat requests hours to show as AM or PM.

That’s pretty much it, except for updating the version number and giving some better names to variables.

To get the latest version, go to the digests page.

Digests 3.2.1-RC8 released

There’s lots of new stuff in this release and lots of bugs fixed. You can download it from GitHub here or from this site here. This version works on phpBB 3.2 Rhea and will not work on phpBB 3.1.

To update, make sure you follow the standard procedures. First disable the last version. If you delete data you will lose all digest subscriptions, so it’s not recommended. Delete the old files in /ext/phpbbservices/digests and replace it with the new files. Then enable the extension again.

Big thanks to all who reported bugs and asked for new features. I would be especially lax if I did not mention my French language translator Bonnaphil who asked lots of questions, did lots of testing and pushed me to deliver features I hadn’t thought of, like making the digests (well, HTML version anyhow) look “prettier”.

I haven’t figured out how to copy the branch to master, so if you know please leave me a note.

Summary of changes, Digests 3.2.0 to 3.2.1

Administration Control Panel

  • Explanation text added for weekly digest day, to indicate that digests may be received on the day before the day set for users in the western hemisphere, since digests are sent based on UTC. This is already stated in the User Control Panel.
  • BEHAVIOR CHANGE: Digests time zone removed. This was used in Edit Subscribers and in Balance load to show the hour digests are sent out. The timezone for the user in the User Control Panel is now used instead. This is less confusing.
  • NEW FEATURE: Switch allows administrator to show the full forum path in digests. If set this applies to all users.
  • NEW FEATURE: Switch allows administrator to lowercase the digest type. This is at the request of a language translator.
  • BUG FIX: The default board language is no longer used as the basis for making digest language decisions in the ACP. Instead, the language you set in the UCP is used instead which is the way phpBB does it elsewhere.
    NEW FEATURE: Dates are translated into the language needed. The PHP date function only works with English formatted dates. strftime and a new function is used to translate date items like month names from English to the language desired.
  • BUG FIX: When jQuery usage was introduced into Digests, it had the effect of undoing logic written that sent to the web server only those variables that were changed on the Edit Subscriber screen on form submission. Since there are so many fields, it could cause a PHP max_input_vars error. Changed the logic so that jQuery will handle this instead of regular Javascript.
  • Cleaned up the jQuery so selectors weren’t used on multiple lines, the preferred way to do things in jQuery.
  • NEW FEATURE: Administrator can specify tags to remove from HTML digests to prevent certain security issues. Removed tags are replaced with a message indicating the tag was removed and suggesting clicking on the post link to see the full content.
  • BUG FIX: Mass subscriptions no longer set all new subscribers to the same hour. Rather if a random hour is requested, one is generated for each new subscriber. Previously after doing this you would also have to run Balance Load to distribute the digest send hour.
  • NEW FEATURE: Balance load shows the average digests sent per hour (rounded to the nearest whole number). Hours are shown in more usable format based on user’s date/time format. A list of subscribers for the hour are also now shown with their digest type annotated by the styling used (none = daily, italics = weekly, bolded = monthly). In addition, you can limit balancing to a digest type (daily, weekly or monthly), you can specify the hours you want to balance and you can specify the hours to be used for balancing (for example, to avoid sending a digest in the middle of the night).
  • User default settings now use the same language strings used in the User Control Panel interface where possible.

General

  • NEW FEATURE: Timezone hours can be prefaced with a + if the hour is positive and the sprintf markup requests it.
  • BUG FIX: Function dateFormatToStrftime will ignore AM/PM designation in date formats.
    Created many additional plural rules for language files were appropriate
  • Where there is a substitution of more than one variable in a language string, they are now marked positionally, ex: %1$d instead of %d. This aids repositioning language elements in non-English translations.
  • BUG FIX: Subscribe to digests on registration logic changed. You must be a human (i.e. omits possibility that bots could get a digest).
  • MAJOR BUG FIX: Migrations of Digests from phpBB 3.0 no longer fails because an instance of the tools factory was not first declared. (This was not necessary in phpBB 3.1.)

Template changes

  • MAJOR ENHANCEMENT: HTML digests look “prettier”. Background colors subtly change between posts within a topic making it easier to distinguish when posts begin and end. Background colors are consistent. Page background is now white. DIV boxes containing text no longer stretch across the width of the window if there is not much content.

Mailer

  • NEW FEATURE: Switch allows administrator to lowercase the first letter of the digest type.
  • NEW FEATURE: If show the forum path is enabled, the forum name in the digest will appear something like Category 1 :: Forum 1 :: Category A :: Forum B. Otherwise it will just show Forum B.
  • NEW FEATURE: If the switch that allows administrator to lowercase the digest type is enabled, behavior changes for how the board name is shown in digests. For example, in certain languages “Daily Digest” would be “Digest Daily” and will logically precede the board name. If set to yes, then the digest title will appear something like “Digest daily of my board name” depending on the translation with the first letter of the board name in lowercase.
  • BUG FIX: If for some reason the last time digests were successfully sent is after the current time, the mailer will log an error and abort. Otherwise an infinite loop starts.
  • BUG FIX: Language files are applied for each user based on their preferred language. This keeps for example a user who has their language set to French from getting English markup in the digest.
  • BUG FIX: Mailer no longer allows bots and inactive users to be selected in the initial query of users receiving digests for an hour.
  • MAJOR BUG FIX: List of posts for the time period is no longer constrained by the SQL call to limit the maximum number of digests based on the administrator or user’s maximum posts settings. This can cause other users to not see posts.
  • MAJOR BUG FIX: Fixed bug that did not correctly compute month start and end dates for monthly digests. Date function was incorrectly called generating a warning.
  • NEW FEATURE: Digest translator can now be optionally identified and linked at the bottom of the digest.
  • BUG FIX: Fixed bug that printed a “No private messages” announcement twice in text digests.
  • MAJOR BUG FIX: Fixed bug that did not properly sort posts for some digests because no array was initialized.
  • NEW FEATURE: Dates appearing in digests are now translated to show month names and days of week (including abbreviations) in the user’s language.
  • NEW FEATURE: Administrator specified tags (which may cause security problems) are removed from HTML posts. A message is substituted letting the subscriber know content was removed.

User Control Panel

  • Spacing for digest type is now consistent with other spacing for fields on the screen.