Month: July 2019

July 2019 work summary

On: By: Mark

My work in July kept me quite busy and was quite profitable. A number of projects for clients started earlier continued in July, but some concluded. It’s hard to remember all the details and clients, so I will be more succinct than usual.

I’ve been getting a lot of Aussie clients lately. Not sure what’s going on. Maybe my name is being passed around by word of mouth. I live in the United States but likely 75% of my work is foreign. It used to be mostly for clients in Europe. Lately it’s been more in Asia: Australia and the Philippines in particular. The time difference is a challenge but it can be an advantage too. The larger, hairy jobs are usually complex upgrades. I can work in the day while the board is offline in the night halfway around the world.

After a slow April and May, I’m now more than caught up with my revenue goal. If the rest of the year continues the way the first seven months have, I should surpass my 2018 revenue by a bit.

  • More work for a client I’ve had for a few years now. Originally I worked on her forum, which is now dormant. Now I mostly help her with hosting and domain issues. I completed the domain registry transfer of one of her domains from GoDaddy. Not sure why it didn’t happen last year as I thought it was taken care of. It’s a .ca domain and they make you wait 60 days between registry transfers. Also, her Let’s Encrypt certificate kept failing. I generated a new one in cPanel on her web host. I also assisted in the procurement of another domain she had. A “friend” of hers controlled the domain and wouldn’t release it. It expired, she bought it, and I redirected it to a new domain.
  • I finally got paid for two reseller commissions by siteground.com. I appreciate clients who use my affiliate link. You don’t pay more and I netted $100.
  • More work for an Aussie client I did work for last month: same domain, just more tweaks. I had it all on a spreadsheet but I deleted the spreadsheet, so I don’t recall the details. But it was a significant amount of work for which I was paid $300.
  • I redid all the work I did for a client last December. It was a strange case of moving and converting a phpBB 2 forum. He had the extract I provided back then, just never got around to rehosting it. Meanwhile, users kept posting on the forum, so I did it again. Client lives in the United Kingdom.
  • Upgraded a forum from phpBB 3.2.4 to 3.2.7. FTP seemed flaky. Also, there was an issue getting into the ACP after the update. I had to upload the media embed extension using the cPanel file manager instead of FTP because it was flakey. I also needed to use cPanel to turn off SuperCache for the logo to reappear.
  • Updated a forum from phpBB 3.2.5 to 3.2.7. No issues. A couple weeks later, I also updated the advertisement management extension.
  • Did something that was actually unusual: an installation of phpBB on a new server. It will be a brand new forum! Client spun up a server just for the forum. Very nice client and paid very well. There was plenty of work aside from just installing phpBB. Work included: long requirements discussion, installing phpBB 3.2.7, resolving some issues with FTP and SSH, working around an iconv() error I’ve encountered before (no real solution has been addressed), creating the requested forums, the requested groups and assigned appropriate forum permissions. Client also paid for three hours of future time when issues or changes are needed.
  • Another huge job, for another Aussie client, that is not quite complete.
    • In part one, I updated their forum from phpBB 3.2.2 to 3.2.7. The major issue is that email notifications exceeded the web host’s quota and buying more email quota is not allowed, so emailing needed to be staggered using a system cron. The mailing program in phpBB, /includes/functions_messenger.php was renamed by web host technical support to stop emails from going out. They looked like spam to the web host. Made a phpBB email package size of 50 because hourly quota is 500 emails, so 50 emails every 5 minutes should keep him under quota. But doing a system cron on Windows proved impossible on this particular web host. I asked my client if there was a compelling reason not to move from a Microsoft server to a Linux server. I should be able to program a system cron there and thus address the staggered emailing issue because the technical limitations won’t exist on the Linux server
    • In part two, the client takes my advice and makes the leap to a Linux server, which meant moving a number of domains, forums, one Wiki and a host of other content including a Joomla front end. Client’s web host was not terribly responsive to his needs, which meant lots of fruitless back and forth emails. The project is mostly done but there are a few loose ends. There is a huge 6GB MySQL database tied to a Wiki that uses Microsoft ASP. Client understands the Wiki won’t be accessible on Linux but doesn’t care too much, just wants to retain the data. It was too big for me to move with the only tool I had (Plesk). Web host tried and failed too. There may be integrity issues with the database. So that’s still going on, and I’m waiting on some information to program the system cron which because I asked is 200 emails per hour per domain, 500 per hour for all domains hosted for the client. But I did move over three phpBB forums including the databases, one Joomla front end and lot of static pages. This seems to keep going on and one, but hopefully I’ll report a successful conclusion in my August report.
  • Troubleshooting on phpBB 3.0.12 forum. The client, an administrator, could not login or access the ACP. I registered as myself with no issues. Via the database, I gave myself founder privileges. But I didn’t see the ACP link. I had to copy the phpbb_users.user_permissions value in the database from an administrator’s permissions to get the ACP link. I changed his password in the ACP and provided an idea of the cost to upgrade to phpBB 3.2.7. That work may show up in a future report.
  • Installed a SSL certificate for domain.
  • A client’s third party script was failing. I thought it was because old mysql driver calls were in the code, but the real problem was that a common include file containing the database access information referenced the database server name and it was not localhost. Made it 127.0.0.1 to match the phpBB config.php file and the scripts started working.
  • Spent about 90 minutes online tutoring a client who had taken over a large forum. The web host is flaky. I showed her how to backup database and files on her hosting. I also made backup of the database and files and posted them to Google Drive. The Network Solutions file manager could not make an archive to contain all the files, so I had to download all the files using FTP and create the archive myself. It turned out to be 16GB, so I paid for extra space on my Google Drive to transfer them to her. My slow Comcast upload speed though meant it took over three hours to upload!
  • Additional labor for a client I did a lot of work for last month. Mostly I answered lots of questions. I created a special report for her using SQL and phpMyAdmin. It shows duplicate posts, many of which are spam posts. Her moderators will go through all 600 of them manually and remove the obviously spammy ones. She wants a clean board again. I also removed a style credit line from footer of her forum.
  • Updated a forum from phpBB 3.2.0 to 3.2.7. The client will take care of updating the prosilver_se style with his changes and one extension.
  • Updated a forum from phpBB 3.2.5 to 3.2.7. The site uses a proprietary SiteSplat style. There was also an error on the manage extensions page. I had to go into the database and remove the bogus extension from the phpbb_ext table, and then purge the cache for the issue to go away. I did not update style or the SiteSplat extension.
  • Updated a forum from phpBB 3.2.5 to 3.2.7. I also updated the following extensions: Advanced BBCode, phpBB Media Embed, Cleantalk and Tapatalk. I disabled the beta Copy new topic extension that I had installed some months ago for the client. No issues, but the proprietary Dreamhost web host control panel took quite a while to puzzle through — I really don’t like it, it’s so confusing! The FTP settings I had no longer works, so I had to figure out correct FTP settings again.

The phpBB file structure

On: By: Mark

I try to write at least one “tip” post a month. Today I want to dig into phpBB’s file structure. Looking at the folder and file names, a lot of it is self explanatory, but not all of it is.

To start, the file structure has two parts: data and software. Most of it is software which is replaced with upgrades or updates. But some of it is pure data and must be carefully avoided when upgrading or updating. Note that most of your content is actually stored in a database attached to your forum.

Data folders

There are three data folders:

  • /files contains any files that are uploaded by users, with a few exceptions. Mostly these are images attached to posts, but sometimes they are documents attached to posts if permissions allow. If you allow attachments to private messages, they are also stored here. If you go into the files folder though, you will see file names that don’t look the least bit like images and lack the file suffix you would expect, such as a .jpg, .gif or .png. The file names look all scrambled, such as “2_09acd47ea11adef5e4d9cac2903e4ddc”. This is mostly a random hexadecimal file name created by the phpBB software. Its purpose is to hide the intent of the file. If a file represents a thumbnail, it is prefixed with thumb_. phpBB uses a database table, the phpbb_attachments table, and a program called download/file.php to actually render the file. file.php sends HTTP headers that tell the browser how to interpret the file, such as a JPEG image. It’s actually quite clever — security through obscurity! Unless you are on the forum reading a topic, you have no idea what the uploaded file name was, its file type, or who uploaded it. Moreover, you can only indirectly download it. You don’t want to tamper with this folder as it must be consistent with the phpbb_attachments table. Since uploaded files must be stored, this folder must have 777 file permissions.
  • /images contains images, which is not too surprising but not any images users attached to posts, as they are in the /files folder. If you drill down into the folder you will see a bunch of subfolders that describe the content of images placed inside them, such as /avatars, /ranks and /smilies. If you allow users to upload an avatar, they are stored in the /images/avatars/upload folder, which is why this folder should have 777 file permissions. If you add rank images or smilies packs, they must be uploaded with FTP into the appropriate folder to be seen and used. In general, you don’t need to mess with this folder. Just leave it alone. But since there is no software inside it, it’s a pure data folder.
  • /store contains miscellaneous files that may be created by phpBB. It is most often used to store an extract of your database, which you can do: ACP > Maintenance > Backup. If you recover your database, it will read the file placed here using the ACP > Maintenance > Restore function. It has occasional other uses. When upgrading or updating phpBB, lock files are created here. If things go awry you may have to manually delete them. Extensions that store data outside of the database may also write data here. These folders must be identified by the vendor and extension name. For example, my digests extension uses a /store/phpbbservices/digests folder.

Styles folder

The /styles folder can be considered a data folder, but only if you made custom changes to the styles. Your custom changes will get overwritten with a phpBB upgrade or update, or if you update a non prosilver style. As a general practice, you should backup all your folders and files before upgrading or updating. But if you made custom changes to a style, you really need to back these up and reapply them if desired after an update or upgrade. You might want to read my post on creating a custom style to minimize the impact of these changes.

Cache folder

The /cache folder hints on its function: it creates a “cache” of PHP scripts that speed up the process of rendering web pages. For example, your style creates a unique look for your site, so parts of the web page are actually a .php program placed here dynamically created by phpBB.

The cache folder is new since phpBB 3.0, but since phpBB 3.2 there are two major subfolders: /cache/production contains your “production” cache and /cache/installer is used during updating or upgrading phpBB. The cache folder itself must have 777 permissions so they can be publicly written to. The same is true with all folders and files inside it: folders must have 777 file permissions and files must have 666 file permissions.

When you click on the “Purge the cache” button in the ACP, you are destroying files in the /cache/production folder. If you accidentally delete files in the cache folder outside of phpBB, they will get recreated. You will notice a little lag rendering these web pages while the cache files are recreated.

Tip: if you get weird error messages that seem to lock up your board, use your FTP program or web host control panel file manager to delete the files and folders in the /cache/production folder. Often, the site will come back up with a page refresh!

config.php file

This is a critical file in your forum’s root folder that is essentially a data file and should not normally be messed with. Why? This joins phpBB software to your database, so it contains your database host server, name and the credentials of the account (including the password!) to your database. Consequently, don’t mess with this file unless you know what you are doing and do not allow it to be publicly readable. Ideally, the file will have 600 file permissions.

Tip: if you move from PHP 5.x to 7.x and you get a message saying mysql drivers are not supported, edit the line in this file:

$dbms = 'mysql';

changing to:

$dbms = 'mysqli';

and your forum should come up! PHP 7 does not support older mysql drivers and requires the newer mysqli (i is for “interactive”) drivers.

Software folders

With the exception of the above files and folders, all the other folders and files are software. I use the term “software” liberally. Technically, only PHP programs and Javascript files are “software” in that they implement programmatic logic. In reality though there are a host of technologies used to render phpBB including HTML, CSS, jQuery and XML. 

For the most part the folder name describes what they do, so I’ll concentrate on important folders.

  • Principle programs. The principle programs are in the forum’s root file. For example, index.php renders the index and viewtopic.php assembles and presents all posts in a topic. Obviously you don’t want to edit any of these.
  • /adm contains the styles used inside the Administration Control Panel. Except for the index.php program, the actual ACP software is in /includes/acp.
  • /ext folder contains extensions. Extensions are additional functionality that developers write that does not come “out of the box” with phpBB. They are placed in an ext/vendor/package folder. For example, my digests extension if installed is in the /ext/phpbbservices/digests. The vendor is phpbbservices and the extension name is digests. These must be uploaded with FTP then enabled: ACP > Customise > Manage extensions
  • /includes contains key support programs used by the principle programs.
  • /language contains various language translations. British English is provided. If you install one or more language packs, they are uploaded to this folder. For example, a French language translation will be in a /language/fr folder.
  • /phpbb contains a set of phpBB objects and services that “object-orient” phpBB. This object-orientation became system-wide starting in phpBB 3.1. Most programs in the /includes folder as well as the key programs use one or more objects or services in these folders.
  • /vendor contains third-party software than phpBB needs to run. Since phpBB 3.1, phpBB has depended on a number of high-quality, 3rd-party software solutions. If you look inside the /vendor folder you can see all of them. For example, phpBB uses the /vendor/twig folder’s software for handing its templates, i.e. inserting dynamic data into web pages.

Hopefully this gives you a good idea of how phpBB’s file organization works.

 

Your web host may be virtual

On: By: Mark

Introduction

I recently wrote a post about putting phpBB in the Google Cloud. I learned that it’s not too hard to do if you have decent technical skills or even if they are more modest. There could be some serious upsides to putting your forum in a cloud like Google’s, Amazon Web Services (AWS) or Microsoft Azure. (There are other cloud vendors out there.) These could include lower costs, higher uptime, and scalability if you forum gets suddenly popular.

Most of us though contract with web hosts. For example, I use Siteground. Web hosts have server rooms somewhere where they keep all the equipment they need to host your forum plus lots of other websites. Most web hosts have multiple server rooms in various countries. The closer these are to their customers and their site viewers, the better. For example, Siteground has server farms in Chicago, London, Amsterdam and Singapore. They have incentive to organize their data centers to be fast and reliable because they control them. Siteground does this not only with four server farms, but by having an end-to-end solid state infrastructure. They figured out that although solid state drives (SSDs) were more expensive, they were heaps more reliable and faster than filling their server rooms with mechanical disk drives. It’s been key to their success as a company.

Virtual hosting

These days though some web hosts are figuring out they don’t need to bother with the actual hosting anymore. There are two ways they do this. One is old, the other is new.

The first way is to be a reseller. For example, ABC Hosting may actually rent servers in (hypothetically speaking) a Rackspace server room. Becoming a reseller is not hard. Siteground will let you be a reseller. Resellers are often people like me who have multiple clients and as a convenience to their customers also provide hosting. I don’t want to bother setting up a server farm, particularly if I can lease one. If I did, I would probably choose to become a Siteground reseller, since Siteground’s spiffy servers sold me on being their client. Siteground would provide a front end console for me to use, and consoles that my customers would use too to which I would apply my own logo and some custom pages. From the customer’s perspective, it looks like I have my own server room. The downside is that I would become responsible for any hosting issues. I would essentially be the support department, and I’m not available 24/7. I don’t want to get involved in the minutia of my client’s hosting, so I don’t expect to ever become a reseller, even though it would generate a good deal of passive income for my business.

The second way is that some web hosts are becoming virtual by using cloud providers. Who’s the number one host on the web? You probably don’t have to think too much: GoDaddy. You may be surprised to learn that in 2018, GoDaddy decided to move much of its hosting inside AWS. You can read why here. Basically, GoDaddy realized that AWS built a much better infrastructure. They can resell Amazon’s cloud services under their own label for less than they can maintain their own hosting centers. AWS has a sophisticated set of services and they have the fast connection and high reliability thing all figured out. This is not good news for GoDaddy’s hosting staff. Presumably most of them will be laid off at some point.

All this suggests that web hosting will be undergoing a fundamental transformation as hosts ditch their own hosting centers to find better reselling deals in the cloud. In short, your web host may become a virtual web host. If you host on GoDaddy, there’s a good chance it’s already virtual hosted on AWS.

Should you host in the cloud?

This does raise the question: why not just buy your hosting from a cloud vendor like AWS and skip a middle man? If you read my posts on cloud hosting, you’ll realize the main issue is that cloud hosting tends to be complicated to set up, maintain and troubleshoot, at least from the perspective of someone trying to get some web space without a lot of technical skills. Virtual web hosts like GoDaddy essentially become front ends for optimizing the hosting experience for people likely a lot like you who want the process to be simpler. So they offer 24/7 support, domain management and basic customer handholding while putting up a virtual front end that suggests they are doing all this themselves when in fact the technical infrastructure is outsourced to a major cloud vendor.

My bet is that at some point Siteground will do the same, in which case I will have less reason to use them. If I know a suite of virtual web hosts are all using AWS, for example, I can get choosier and choose a virtual host based on their support and the ease by which I can do things via their control panels. I can assume the reliability and speed will all be excellent since they are hosted in a professionally managed commercial cloud. Since I do have the technical skills to put my sites in a cloud like AWS, at some point I will probably just do that. I pay a premium primarily to call someone on the phone to resolve some technical issues. Right now the $20/month I pay for Siteground hosting for my domains is reasonable, even though I am guessing I could pay $10/month or less putting my sites in the cloud. I’d just have to fix any technical problems myself, and right now the cost difference doesn’t make it worth my time.

For most of you, this is probably true too. Price is certainly important when you decide who to host with, but ready support, easy interfaces to managing your sites and fast page load speeds probably matter more. At some point you either won’t know or won’t care if your web sites are actually in a major cloud vendor’s facilities somewhere. Virtual web hosts aren’t probably going to advertise this either.

If interested in Siteground hosting, use my affiliate link

If you are intrigued about my discussion of Siteground for web hosting, learn more on my rehosting page. If you decide to host with Siteground, please use my affiliate link. You won’t pay anything extra and I will earn a small commission.

June 2019 work summary

On: By: Mark

My work in April and May was surreally slow. Happily, this was not the case in June. Work just kept coming in and it got a little frantic at times. A lot of it was hairy upgrades from phpBB 3.0, something of a speciality of mine. Once I get paid for all my work, I should reach my midyear revenue goal which if it repeats for the second half of the year should top last year’s income, then an all time high.

  • The original issue was blank screen that users were having accessing the forum. It was some sort of security issue. Someone, probably the web host, placed a line at the top of the forum’s .htaccess file that triggered the issue. Commenting out the first line of the file brought the board up. Next I was asked to upgrade the from from phpBB 3.0.11 to 3.2.7. I completed the upgrade completed with no issues. I installed the prosilver_se style, but made a number of tweaks to remove rounded corners, apply their logo across the entire background, put black along the sides of document window and to add two links to the navigation bar. I then installed an unapproved version of the Board3 Portal extension that seemed to work with phpBB 3.2 correctly. I edited the .htaccess file so portal page came up first. The portal though was not quite ready for prime time. Time showed an issue with adding new portal modules, so I was asked to disable it.
  • I updated a forum from phpBB 3.2.2 to 3.2.7. I reinstalled an updated version of the prosilver_se style and their logo. I added the ACP Add User extension, which allows administrators to create new users. This is a popular extension for locked down boards where registration is disallowed. I tried to upgrade PHP to 7.2 on Bluehost but it didn’t take. I suggested the client file a ticket to get it properly upgraded. It took a few days but Bluehost finally came through.
  • Troubleshooting for a long-existing client’s forum. Negative values for the number of unread of unread private messages were appearing on the navigation bar, possibly due to a bug in my digests extension. I created some SQL that addressed the problem, and implemented it on both their development and production boards. I then upgraded both the development and production forums from phpBB 3.2.5 to 3.2.7. I also upgraded my digests extension on both forums from 3.2.10 to 3.2.13 and applied a patch that should prevent negative values from being stored.
  • I updated a forum from phpBB 3.2.4 to 3.2.7. I also updated the board’s Latte style, being careful not to overwrite changes and to recompile the style (it uses .sccs files). I updated my digests extension from version 3.2.10 to 3.2.13 and the Media embed extension to version 1.1.0. Later in the month there was additional work. Emails wouldn’t go out; they were being captured by a web host’s outgoing email spam filter. I chatted with the web host. They told me to change the email server name and email started working again. I disabled the user control panel interface again for digests as the client wanted to be the only subscriber.
  • I upgraded a forum from phpBB 3.0.12 to 3.2.7. I used a standard prosilver style installation with a logo. The forum has about 100,000 posts and there were no mods to worry about. It took about an hour to do the upgrade but there were no issues during the upgrade. I disabled the contact form. I enabled the reCaptcha V2 spambot countermeasure. I added a home link and placed their logo in the header. I made a test post to make sure it was working well. I suggested changing PHP to 7.2 and see if other software on the site crashed and if not leave it at PHP 7.2. PHP 5.6 is currently being used.
  • Big project #1. The forum required an upgrade from phpBB 3.0.14 to 3.2.7 but had to be rehosted at the same time. This is because the content on the forum is sensitive in the country where it is used. It was reported and the web host (FatCow BTW) just shut it down. Fortunately, they were able to get the database and files ported over to the new host. There I created a test upgraded version of the forum and did my best to replicate the many features and unique look of the forum on phpBB 3.2.7. Fortunately the existing metrolike style was available and most of the modifications on the board for phpBB 3.0 were available for phpBB 3.2 as extension. The number of extensions desired though was highly unusual and was in the dozens, and not all were approved. Invariably, a few modifications did not have equivalent extensions, so some things required manual changes. For example, there is a unique message that shows to new users inviting them to register that was not able to be done with an extension, primarily because it had to show only to guests. I added this manually into the forum’s header with template logic to show only to guests. In addition, the client wanted forum permissions changed, keeping everyone from reading forums with sensitive content but allowing search indexes to still index its content. I did this by creating a special group and giving read permissions only to people in this special group. It took a lot of back and forth and since the client was nervous it was done via email. I had hoped to use Skype to work through the myriad questions. I was paid for the analysis and waited for permission to do the formal upgrade, which I got yesterday and finished. The client is now reviewing it.
  • Big project #2. This was a lot like the last big project in many ways, except the forum does not contain sensitive content, but is a big and very successful forum based in Australia. The board was a heavily modded phpBB 3.0.5 installation that needed to be completely migrated to phpBB 3.2.7. I began with a test installation of phpBB 3.2.7 to try styling and extensions. It soon became clear that we needed a full test upgrade, upgrading a copy of the whole production database, not to mention more than 100,000 image files. There were so many images that the client ran out of hosting space. So I solved it with a symbolic link to the old files folder. phpBB Gallery was a big part of this upgrade and had to move successfully. Unfortunately, the version of phpBB Gallery for phpBB 3.2 is not approved. The original developer abandoned it, someone else took it up and it was buggy. So there was a lot of trial and error to get not just attached images but phpBB Gallery to work acceptably. It was apparently too big a project for the last developer who tried to do this upgrade. He left due to mental health issues. (It’s unclear if it was related to this work!) The full test upgrade revealed a host of challenges. For example, I ended up manually adding and removing columns from phpBB Gallery tables to port the data over successfully and get rid of a lot of errors. There were also logout issues on the existing forum that emerged, issues with the upgrade itself where we ran out of memory (on virtual hosting!), making everything quite challenging. There were also issue with the Javascript lightbox integration. I’m happy to say that it all went well thanks to the full trial and a lot of beta testing by the board admin and his beta testing team. Still finishing some of the final tweaks and there may be more work to add security certificates and upgrade PHP. There are other applications on the domain that apparently aren’t PHP 7 compatible.
  • Updated a forum for an existing client from phpBB 3.2.5 to 3.2.7. Updated prosilver_se style and language pack (Danish) as well.
  • Maintenance work for an existing forum for medical professionals. I’ve had them as a client for nearly ten years now! I added and configured the Board Rules extension using text provided by the client. It acts as a disclaimer of sorts. I made signing off on it to be required by new users. In Plesk, I updated 20 packages that needed updating. Later, I updated phpBB from version 3.2.3 to 3.2.7, updated the phpBB Media Embed and my digests extension. I updated four of the 5 styles in use because they had newer versions.
  • Urgent request from another client in Australia. My Smartfeed (version 3.2.10) extension would not run. I was given SSH access but didn’t have much in the way of clues in the error log. Eventually I just disabled the extension, then enabled it again and it worked.