Smartfeed 3.0.3 RC2 Released

The main change in this release was to add a missing join that was causing a Cartesian product, but also changed some code to help the mod get approved. I am submitting this release candidate to the phpBB extension team for review and possible approval. It’s pretty unusual to get it approved on the first pass, but you never know.

It can be downloaded here.

How to reset group and user forum permissions for phpBB forums

phpBB has an awesome permissions system that is very fine grained and very comprehensive. While very functional it can often be hard to use too because there are so many possible permutations: group permissions, user permissions, role permissions, etc.

It’s not hard at all if you have a board with lots of forums and/or lots of groups to find yourself in forum permissions hell. A user might belong to multiple groups, for example. Which group permissions will override other group permissions? You might long for a button that just resets all the forum permissions, but nothing like this exists. You may want to comprehensively reset all your forum permissions instead.

Group forum permissions and user forum permissions allow you to finely tune these permissions. With lots of forums though the user interface can be overwhelming. It can be hard to click on the drop down controls to set them properly and to do this for all the forum permissions for one group. Worse, sometimes you’ll take the time to do this and you’ll find out not all the permissions took. (An aside: if this happens to you, it is probably because there are so many fields on the form you are submitting that PHP can’t handle it all, but won’t tell you about it. Upping your PHP max_input_vars setting may solve this problem. Or you can do it to a subset of forums at a time, rather than all.)

If you want to reset your forum permissions, try these steps:

  1. Backup the database first. You may make mistakes or you may find you don’t want to do this when you see the result.
  2. Create a forum called something like “No privileges forum”. Do not copy forum permissions. Note: if you do this and look at the permissions for a user or group for this forum, you’ll see “No role assigned”. This is not the same as no permissions at all.
  3. If you want to get rid of some of your user defined user groups, now is a good time to blow them away. It will make the rest of this easier.
  4. ACP > Permissions > Group forum permissions. For each predefined group, set the forum permission for the “No privileges forum” to “No access” and save.
  5. For each user defined group, do the same thing.
  6. Now copy these permissions to your other forums. ACP > Forums > Copy forum permissions. For the first field select the “No privileges forum” forum. For the Apply permissions to field, select all the other forums. Warning: at the end of this step, no one will be able to access any forums on the main index. In fact, no forums will show and you should see “This board has no forums.” This includes administrators. Don’t worry. The forums are still there just not accessible. The message is really incorrect, but presumably set this way to deter spammers.
  7. Now change the permissions for one forum (but not the “No privileges forum”) so that the group privileges are the way you want them for all your groups. This means selecting the forum role to apply for each group or (not advised) using the Advanced Permissions link if you need very granular permissions. (It is better to change the forum role permissions instead.) If you don’t set the privilege for a group they will have the No access role because of the previous step: the forum will not be seen on the index at all for that group (unless belonging to another group gives them that privilege). Start with the predefined groups then do any user defined groups. You will have to do this for each group of interest but always to the same forum.
  8. Now do step 6 again but in the first field select the forum whose privileges you just granted and apply it to all the other forums, except the “No privileges forum”.
  9. If you have user defined groups that should access only certain forums, set the forum privileges using ACP > Users and groups > Group forum permissions. Do this for each group. Again, it’s advisable to use a forum role.
  10. All user (non-group) privileges are now lost. If you need to add privileges for specific users outside of a group, do this now. It’s a best practice to not set individual privileges, but grant all privileges through groups.

Test. You can test as yourself or test using a user’s permissions, but all should be well.

Digests 3.0.0 (Alpha) released

Phew! What a journey. Obviously I got over the hurdle in my last post. The first version of this extension, 3.0.0 (Alpha) is complete.

You can find download and installation instructions on the phpBB topic.

Please do not use in a production environment. Provide any feedback you have on the phpBB topic, not here please.

My thanks to the phpBB Extensions Writers for their extensive help and coaching through this challenging process.

Running as cron issues with digests extension

The good news is that the class I created to override the messenger class is written and working correctly. The last issue is to get the mailer running as a phpBB cron task. I am currently stuck there and am seeking assistance to get past this last hurdle.

This latest ripple is discussed here.

It’s pretty mysterious and hard to figure out. TWIG is an open source replacement for phpBB’s templating system. phpBB calls it and TWIG generates the templates. The templates are needed to do the work of formatting the digests in the email. It works fine when I run it in manual mode but when I run it as a cron task it fails with the error described.

It may be that phpBB’s cron system assumes the templating system is not needed, which means I will have to add it. More likely it is something obscure about TWIG and I need a TWIG expert to suggest a fix/workaround.

Hard to know how long this will take. It may be something that is figured out soon, or not. Fixing this problem may expose others. It’s a critical component, however, so I can’t release without it.

Digests extension status report

In my last post on this topic, I outlined what I had left to do to put out a first release of a digests extension. Update below on each point:

  • Thoroughly testing the new way of sending digests via phpBB system cron. In creating a manual mailer and integrating it into the Administration Control Panel I have in effect created an easy to use test tool for mailing digests. So far though I haven’t actually emailed anything, but I can write individual digests to the digests’ cache folder. This has a number of advantages, the biggest one being that I can bring it up in a browser and carefully examine it for errors. Once inside a browser I can closely inspect it with Firebug and run HTML validation tests too. The digest extension will send HTML digests as HTML 5 instead of XHMTL 1.1, which is cleaner code. Anyhow, it’s much easier to test all the permutations with the manual mailer and I thoroughly went through testing all the features of the mailer except actually emailing the digest. I can’t get my local instance of XAMPP to send emails. To test that part I will do it on my website where that integration is built in. It’s the last step in development.
  • Completing the integration of all the features with the manual mailer and the digest cron job. Done.
  • Extending the messenger class to send HTML emails. Last piece of functionality to port and I’ll do that part on my web server (where I can send actual emails) and port those changes back to my local instance for packaging.
  • General code review and cleanup. I’ve cleaned up all the language variables and removed dozens of them that were no longer used. I’ve done a pass through all the major blocks of code but it will need a couple more passes as part of the code are unnecessarily complicated.
  • Updating the migration program to properly migrate digest settings from the 3.0 mod to the 3.1 extension. Still to do. This could potentially be time consuming. The goal is to support the mod from version 2.2.6 forward, but if that’s too complex I might have to require a later version, perhaps the last approved version of the mod.
  • Updating documentation including how to set up a system cron. To do as one of the last steps.

There are other things to do:

  • Verify that all the functionality for the file changes in the mod are incorporated in the extension. This is virtually done but there are a few controls and such that won’t be in the extension, mainly because the new architecture makes them moot. I had about a week of delay trying to get the interface where you can sign up for a digest upon registration working. I had to figure out how phpBB 3.1’s event architecture worked, find the right events and tweak the code to work with the event logic.
  • Thoroughly test the actual emailing part. This includes testing using a manual mailing and testing using a scheduled mailing by creating a system cron on my web server and making sure digests actually go out hourly.
  • Full system test. This involves creating a test instance and installing it with phpBB 3.1.6 and seeing if anything breaks. I won’t test all the permutations but try most of the major ones. There are usually a few things that need to be fixed and retested.
  • Publish the code to GitHub.
  • Package it and place it on this server, and link to it.
  • Place appropriate announcements on phpbb.com.

This will probably go out as an Alpha release, as it is well tested and having done so much work on the Smartfeed extension the code is much cleaner than my initial attempt for that extension.

I’m hoping the rest will take about two weeks or less but it depends in part on any business that comes in my mailbox.

December 2015 work summary

Here’s a summary of my work in December 2015, which was a bit lighter than normal. However, I still met my income goals for the year. When I was not fixing user problems I was mostly working on my digests extension.

  • A forum stopped functioning because the sessions table needed repair and the client had no idea how to do this. The more general problem was the storage engine for all the forum’s tables were MyISAM, so to keep the problem from recurring I changed the storage engine for all tables to INNODB, which is what phpBB does for new installations of phpBB. Later, a problem was reported that a newly registered user could not see any forums. After an investigation, I discovered there were no read or post permissions for members of this group for any forum.I tried to add them in phpBB but they didn’t take. An invalid form message occurred instead. Applying permissions for all forums did not work reliably. However, with perseverance I was able to get it to work for a category. I asked customer for guidance on what forums should be exposed to newly registered users and what their privileges should be. Also did an analysis of their spam problem and asked for authorization to fix it.
  • Client had issues when he cleared his forum’s cache. Images no longer appeared because images were no longer on his file system and he didn’t have copies of them. Client decided the best approach was to upgrade from 3.0.12 to 3.1.6. Client’s setup was peculiar because he had a dedicated host with a highly non-standard web host control panel. After a few hours of research I was able to puzzle through his hosting interface and I upgraded his forum from from 3.0.12 to 3.1.6 using the default prosilver style. I had to tweak his php.ini file to remove errors that showed up on the Administration Control Panel’s main screen. I provided client with rehosting advice since he was paying about $500 a month for a dedicated server he didn’t need. When he is ready to rehost I expect to help him.
  • I ghostwrote a technical article for a client.
  • I placed a new logo on the header of a forum, changed the styling to remove text appearing on the header and remove a blue vertical image. At client’s request, I also placed a logo on the login page below the login button. Later in the month I provided advice on her overall site architecture, which was not mobile-friendly. I suggested she move the non-forum content into WordPress, and suggested some shopping cart WordPress plugins that she could use for her electronic commerce. 
  • Client did his own rehosting but afterward the database kept disappearing and database was sometimes empty. His config.php file was pointing to the wrong database and the host was periodically recovering it, causing the weird symptoms. The database user attached to the forum’s database had disappeared, so I created a new database user, gave it needed permissions to the database and changed his config.php accordingly. I restored his database from a file backup. The program viewtopic.php was missing but I uploaded that from a phpBB 3.0.11 reference.
  • Fixed an issue with “Strict Standards: Non-static method utf_normalizer::nfkc() should not be called statically” error message using solution I found here. Forum was using phpBB 3.0.10. After discussion with the client about whether to upgrade to 3.1 or the latest version of 3.0, he decided to upgrade to 3.0.14. I upgraded him and reinstalled his 4 mods. I also upgraded my digests mod that he had installed from version 2.2.11 to 2.2.26. In addition, I tweaked the logo by putting it into the theme and placing an off-white color behind it. I provided guidance on properly using the Advertising Management mod. Later, at client’s request I made tweaks to the banner background image. A couple of days later client discovered his ads were not appearing. I had forgotten to make changes to his style for ad placement. Puzzled through them and had to tweak the style a bit to make them look acceptable.
  • Attended an hour-long Skype call (requirements discussion) with a client. At her request I did some minor editing of various web pages (outside of phpBB). I removed a Facebook icon, changed some link destinations and installed ACP Add User extension on her forum. This was because her forum is very sensitive and the client wanted her staff to create all users manually. I spent another 1.5 hours tutoring a staff member on how to do administration and moderation. Later in the month, I added the ACP Add Users extension to their sandbox forum.
  • I installed the Cleantalk extension on a forum, added the registration key, removed lots of inactive (spam) users and set up the Administration Control Panel for new users to confirm registration so admin didn’t have to approve these personally.
  • Client had changed the database password and updated his config.php but no one could access the forum. This was because the old password was actually embedded in PHP code in the forum’s cache folder. I cleared cache manually using FTP and the board was usable again. I provided instructions on how to do this manually should it recur in the future.

Digests extension update

I’m slogging through the conversion of mail_digests.php, which is now a run method on a mailer class in a cron folder of the extension. There are a couple of issues that are hard to puzzle through. First, I want the same program to be used both as a phpBB system cron (called once an hour) and be called manually from the Administration Control Panel when needed for troubleshooting purposes. (Previously you had to run mail_digests.php from a URL for initial testing. This won’t work with the new architecture.) Figuring out how to do this dual handshake though it pretty challenging.

Digests uses the messenger class to mail digests but the messenger class templating system is not smart enough to do loops, so I have to borrow from the templating system for the bulk of each digest’s content then copy and paste the results into the messenger class. I need loops in the template system to logically arrange posts into blocks of posts inside forums and topics. However, with the new architecture it’s not easy for an ACP program to use templates outside of an ACP environment. I eventually figured out that just copying these templates inside the ACP template area (ext/phpbbservices/digests/acp/style) worked when in an ACP mode.

Now I am focusing on how to save a digest to a file. So far I haven’t figured out how to do this, because the messenger class does not have an assign_display method like the template class has. I’m hoping the phpBB mod writers group can suggest something. It’s not a critical feature but I’d hate to remove it. I’ll most likely have to extend the messenger class to do this. Speaking of extending classes, I have to extend this class anyhow to allow it to send HTML formatted emails. I haven’t extended a class in any major way before, so that’s a learning curve I’ll be tackling soon.

The good news is that all the code from mail_digests.php is ported over. I am tweaking it so it can be used from the ACP. It will need extensive testing which will take time but to do this testing I have to send actual emails with digests out. And that’s a problem because I am doing development in a local environment. I can’t get XAMPP to send email. I tried configuring my development instance to send email via GMail and its SMTP server, but it won’t work. I think my iMac is blocking outgoing traffic and that’s the issue, but I can’t find anything on this with a web search. At some point I’m going to have to move it to my web host and finish development there and that will take some time. I can only go so far before I have to test actual emailing.

So it’s going about as I expected. My consulting work is light at the moment which helps. I’m guessing there is about a month more work before I have a development candidate to put out there for comment, but this may be optimistic. The major blocks of work remaining are:

  • Thoroughly testing the new way of sending digests via phpBB system cron
  • Completing the integration of all the features with the manual mailer and the digest cron job
  • Extending the messenger class to send HTML emails
  • General code review and cleanup
  • Updating the migration program to properly migrate digest settings from the 3.0 mod to the 3.1 extension
  • Updating documentation including how to set up a system cron