Digests extension update: new manually run the mailer interface

The new digests extension will include an ACP interface that will allow the admin to manually run digests. Previously this had to be done by specifying a URL. Mostly it is for initial testing of digests or for occasional troubleshooting. However, you will also use it to manually send digests for days and hours that might have been missed, such as when the cron job was down, using the option to create digests in the past. I have the front end done and am working on the backend code. Take a look:

Manually run digests mailer
Manually run digests mailer

This option will also let you write digests to files on your server to see them via a URL as HTML or text. They will hide in the extension’s cache folder. This potentially allows a static digest to be created showing, say, all public posts on the forum for a particular day or week. Such an interface might get more thoroughly thought through later. Right now this is mostly for testing and troubleshooting. Email clients can ignore HTML markup, but by writing digests to a file you can see it in a browser. Of course writing it to the extension’s cache folder also makes potentially private content publicly accessible, which is why the interface will have a way to clear this cache folder too.

On the backend, since mailing will be done via phpBB’s cron system, mail_digests.php goes away as a standalone program. Instead it will be invoked through a run method on a mailer class for the extension, which will hide in the extension’s cron/task folder.

Digests extension user control panel interface developed

I have replicated the digests modification user control panel interface for my digests extension under development. It all looks and behaves the same as it does for the modification. Fortunately a lot of the code could be copied and pasted with some changes for the new architecture. The hardest part was creating the interface in the first place, as the methods for doing so are not well documented. I spent a lot of time on issues like getting form keys to show as hidden fields. Fortunately, the people contributing to the phpBB extension writers forum were of great help. A snapshot of the Forums Selection screen is shown below.

Forums selection, Digests extension UCP interface
Forums selection, Digests extension UCP interface

With the Administration Control Panel digests extension user interface also complete, the next step is to port mail_digests.php, which as the name implies actually sends the digest. This will include challenges too including:

  • I will need to extend the mailer class to allow HTML formatted emails. I’ve never done this before.
  • It’s not advisable to print raw text to the screen anymore, so I’ll need to develop some sort of template that shows progress as digests are created and mailed.  Most of the time this won’t be seen as digests are run automatically.
  • Add logic to hook mail_digests into phpBB’s cron structure, as it will kick off digests.

So it’s fairly challenging work but a lot of the code can be ported. It’s the stuff above that will consume most of the time and puzzling.

Changing your phpBB tables’ storage engine

Most phpBB forums use the MySQL database to hold the forum’s data. It’s an obvious choice because community editions are free and it comes bundled with your hosting package. When MySQL was first created, there were few storage engines available. Storage engines contain the code (software) and internal data structure for storing data in tables. Consequently unless you have created a new forum it’s likely that your forum’s tables use the old MYISAM storage engine.

With phpBB 3.1, the default storage engine for phpBB tables using MySQL has changed from MYISAM to INNODB. However, the conversion program from 3.0 to 3.1 will not change the storage engine. INNODB is a different way of storing the data in a table. It’s generally more self sufficient than MYISAM and you are less likely to need to repair it. INNODB tables are best used for tables that are frequently read and written to. In fact, the REPAIR command does not work on tables using the INNODB engine.

I generally recommend to my clients to change their tables’ storage engines to INNODB because they are less likely to have issues. I see my clients that struggle through issues like the sessions table needs to be repaired and they don’t know how to do it. Sometimes tables using MYISAM just get inconsistent leading to issues such as orphaned posts.

phpMyAdmin is a tool generally provided by your web host in its control panel. You can use phpMyAdmin to change your storage engine but you can also do it from SSH with the right privileges by running the mysql command. While you can change your storage engine in phpMyAdmin, it’s not intuitive. There is nothing in the software that allows you to do this in the user interface. You can do it on the SQL tab, however.

A few caveats:

  • Backup your database first.
  • You might want to disable your forum during the process. Users might get error messages or timeouts while this is happening.
  • Make sure you know the correct table prefix for your forum’s tables. You can find this in your config.php file in the forum’s root directory. It is usually phpbb_.
  • The more posts and users you have the longer it will take. Generally it takes only a few minutes. It’s possible that phpMyAdmin will time out. In this case go back in when you can, see which tables have been converted and convert any remaining tables.
  • You might want to convert a block of tables at a time, perhaps ten at a time, so you can get feedback that things are proceeding correctly.
  • In phpBB 3.1 there are two CAPTCHA tables that you may not have. These tables are created dynamically if you decide to ask questions on registration. If you don’t have these tables, attempts to change their storage engine will obviously trigger an error.
  • If your sessions table uses a MEMORY storage engine, this is fine. You might want to use MEMORY for the sessions table only because the sessions table is accessed very frequently. The MEMORY storage engine very fast because nothing is stored to a file, but if the database server crashes or is rebooted all the rows in MEMORY tables are lost. This is okay in the case of the sessions table. It just means that people have to log in again.

These instructions assume you are using phpMyAdmin and are running phpBB 3.1 or phpBB 3.0. It’s possible that there will be additional tables. These tables if any are likely from any extensions installed (phpBB 3.1) or modifications installed (phpBB 3.0).

  1. Log into phpMyAdmin
  2. Select the database containing your forum
  3. You can see your tables’ storage engine in the TYPE column. If you are already using INNODB for all your tables there is no point in doing anything.
  4. Click on the SQL tab.
  5. Copy and paste the SQL below into the window. You may first need to edit the SQL to:
    1. Remove tables you don’t have
    2. Remove tables that are already using INNODB for the storage engine
    3. To change the table prefix from phpbb_
  6. When finished verify it worked correctly by reexamining the TYPE column.

The SQL is below the fold: Continue reading “Changing your phpBB tables’ storage engine”

November 2015 work summary

Consulting slowed a bit this month, which was good in a way. It allowed me to work on a beta version of my Smartfeed extension and toward the end of the month I started work on the Digests extension. As always, client information is anonymized. Here’s some of the phpBB work I did in November:

  • I continued ad hoc work for an existing client during the month. I enabled the Windows media extension group so various Windows media would play. First I had to fix another SQLServer CAST bug in phpBB that manifested when enabling the new extension group. The basic issue here is that the client uploads inhouse videos and attaches them to posts, but each video is very large (megabytes). phpBB keeps a configuration variable containing the size of all the items in the files directory. When this value is stored, SQLServer (the client’s database) can’t handle it unless the value is CAST as an integer using the SQL CAST function, which required a code patch that I added. I reported the additional code that is needed as an issue in the phpBB bug tracker. I also participated in an email discussion on the most efficient way to embed videos on the forum and recommended a HTML 5 approach (upload .mp4 files), which phpBB does not currently support. phpBB is behind on integrating HTML 5 videos smartly. I added this issue in the phpBB bug tracker. The template attachment.html needs to be updated to allow the browser to natively play .mp4 videos inside the new HTML 5 <video> tag so no plug in is required. Right now the Quicktime player is loaded, which means it has to be added as a browser plug in, and this is slow and doesn’t usually work on mobile devices. Also provided guidance on Google Analytics: getting an ID for tracking and setting it up. I answered a question about why a user was not able to download an attachment. I also monitored and replied to posts on their forum.
  • I converted a forum from 3.0.12 to 3.1.6 using the prosilver style. I installed the Tapatalk extension and installed and configured the Advertising Management extension, which required renaming a table to port ads from the 3.0 mod to a place where the extension could find them.
  • For one client I installed a new logo with an orange theme. Later in the month I placed a new ad for him and put it above posts in the view topic page. Later in the month I replaced the content of an ad. I investigated issue of a missing user. Apparently his account was deleted. I coached customer on how to find a good web host. The web host saw bad data in private message table in the message_text column. I repaired the table but when it recurred I then changed the storage engine for all tables except the sessions table set to memory to INNODB. So far this is working.
  • I upgraded a forum from 3.0.11 to 3.1.6. I installed the subsilver2 style and made it the default. I integrated the old logo. I provided guidance on using phpBB’s built in ATOM feed and suggested that the user use it to dynamically serve forum topics on his main web site. I provided a link to a phpBB knowledge base article on using phpBB’s feed and showed him some WordPress plug ins that had a feed widget that could consume the feed.
  • I upgraded forum from 3.0.9 to 3.1.6, using prosilver as the default style. Customer agreed to handle the styling issues. When I ran database_update.php, I encountered lots of errors that I had to address, the origins of which are unclear, which meant fixing data and trying again. To get a clean conversion I eventually emptied the migrations table. I still had to do some things manually. This included commenting out code to move the prune users module and emptying then recreating the data in the modules table because module names were not correct. Later in the month the user reported some more issues. After investigation I determined that an acl_option permission that the converter should have added must have missed. Without it a user could not change their profile. I added it and assigned it to 5 roles per my reference database and that solved that problem. A couple of days later another permission issue was discovered when the extensions tab would not appear in the Administration Control Panel. This was also due to a missing permission that I had to add manually. Then the tab reappeared. I double checked all the entries in the configuration table and all the permissions to make sure they were all there compared to a reference. They appear to be so I think there won’t be future problems like this.
  • I placed an additional ad for a client in the Advertising Management extension.
  • To address a serious spam problem, I installed the Cleantalk mod on phpBB 3.0.12. Client must pay $8 for 1 year of service.
  • A HTTP 500 error occurred only on registration. This was due to a bad configuration value for the CAPTCHA (wasn’t changed in the migration from 3.0 to 3.1 and needed the namespace). Post counts were also zeroed on the index page. I manually fixed index to show correct # of posts, last post time and last poster using various SQL queries. I added missing columns in forums table and missing rows in config table that phpBB 3.1 needs. I checked all the tables with a reference to make sure all columns were present for 3.1.
  • After rehosting mysterious HTTP 500 errors were generated on the new host and there was nothing in the error log to provide any clues about what went wrong. I eventually figured out that PHP was not compiled with an integration with the MySQL database. To connect PHP and MySQL, I used Unix yum command with the right integration package and the forum started working. Had to use SSH to do this work as the client had his own server and acted as its tech.
  • A user who upgraded his board for some reason was unable to create a new forum inside it. After investigating, I determined that a column in the forums table did not have a default value. I gave it a null value as a default. Once things were stable I upgraded the forum from 3.1.5 to 3.1.6.
  • I upgraded forum from 3.0.12 to 3.1.6. The old forum had lots of mods but none were critical. Changed style with upgrade from subsilver2 to prosilver.