Hosting migration helpdesk blues

Web development is something I've been doing for the past 15 years, and it's been my day job for the past 8 years. I've also done quite a lot of server administration tasks since then, even if that wasn't part of my actual job description. While there's probably a good economic case for outsourcing hosting and support duties, it can get to be quite frustrating when things simply don't work as intended. This is even more the case when you're a more technically savvy user (and I count myself among those, even if I'm no guru) and you have to deal with a poor hapless tech. You know what it is needs to be done and they just don't seem to be able to do it.

Cartoon of frustrated computer user
Frustrated © 2006 by jonwatson

One of my company's customers had gone belly up and was bought by another party. The new owners wanted to have the old company's website moved off our servers to their own hosting partner. I got the unpleasant job of carrying out the migration. Seeing that the new hosting partner uses Windows servers (whereas we use FreeBSD) already filled me with dread, as there's bound to be compatibility errors. Still, I have no say in the matter, so it's up to me to make it "just work". Here's all the stuff that's gone wrong.

Incorrect FTP password given
Incorrect IP address set for the database server
Incorrect configuration for the database
Incorrect configuration for the webserver

The incorrect password isn't much of a problem (as I could easily set a new one using the web admin panel for which I did have the correct password), but it illustrates the messy situation rather nicely.

Database server trouble

Incorrect IP address

The hosting company uses several database servers, offering a choice between MySQL 4, MySQL 5.5, MS-SQL Server 2008 and MS-Access (yes, Access... I kid you not). Using the web admin panel, I created a new MySQL5 database and associated user, which seemed to work. Unfortunately, while the DNS zone for the account is populated with an entry for mysql5.example.com through which the phpMyAdmin tool is available and defining the IP address through which the web server should connect to the database, this wasn't actually the right address. It turns out they first set the IP address in the DNS zone file and then randomly select a server from their database servers and create the database on that. These settings did, of course, not match. When this happened, on a Thursday morning, I first contacted the helpdesk. It took about 4 hours to coordinate with the new owners of the website that I should have authorisation to actually open a ticket. Then, it took them until Monday to figure out that, in fact, the IP address was set incorrectly and to give me the correct one (which I then had to enter into the DNS zone file manually).

Database configuration

Ok, now that I know where the database is, let's see about accessing it.


<?php

    $database_host = '...';

    $database_user = '...';

    $database_pass = '...';

    if (!mysql_connect ($database_host, $database_user, $database_pass)) echo mysql_error ();

?>

Yep, an error... "mysqlnd cannot connect to MySQL 4.1+ using old authentication". A quick search on the web tells me that PHP 5.3 does not support the old, insecure, 16-character password hashes that were in use prior to MySQL 4.1. The database server, though, has been set up to use only those old hashes. Effectively this means that no PHP 5.3-based website at this hosting company can connect to the MySQL5 server at all.

Of course, the tech on the other end of the line insisted that it was because the password itself was "too weak" and proceeded to change it, unasked, to something like @n3dbK28b!01000. Sure, a better password, but completely unrelated to the problem and still no solution. Go ahead and try this on your database server...

SELECT PASSWORD('@n3dbK28b!01000'), OLD_PASSWORD('@n3dbK28b!01000');

On a properly configured database server, the response will be:

PASSWORD('@n3dbK28b!01000')	OLD_PASSWORD('@n3dbK28b!01000')
*2BAE09CBA796B532701664384A3CE50A5927CAED	48e1302e7f802c84

On an incorrectly configured one, the result will be:

PASSWORD('@n3dbK28b!01000')	OLD_PASSWORD('@n3dbK28b!01000')
48e1302e7f802c84	48e1302e7f802c84

Guess which one I got... The solution to this is normally to change the webserver configuration file my.cnf (on Windows: my.ini) to have the line old_passwords=0 in it. Of course, the tech simply did not believe me and maintains, to this day, that this problem is because my password is too weak. As a user, not an administrator, I can obviously not change the server configuration, so I got around this by using the phpMyAdmin tool to set the old_passwords variable for just that one connection and update the password to the new hash. This will work for now, but the moment somebody tries to change the password again (perhaps this hapless tech?), all will stop working. Here's how to force the new hash:

SET SESSION old_passwords=FALSE;
SET PASSWORD=PASSWORD('@n3dbK28b!01000');

I gave up trying to get the hosting company to adjust their settings, as it took the better part of a day just trying, in vain, to convince the tech that there even is a problem.

Webserver configuration

.htaccess files

The site in question is based on WordPress and phpBB. These are built for Apache, not IIS and rely on the use of .htaccess files. The rewrite rules in particular make use of Apache's mod_rewrite. Fortunately, according to their knowledge base, the hosting company had installed on their IIS servers certain modules which enable the use of Apache-style .htaccess files and enabled them for all site. Unfortunately, it just didn't work. Having access to quite a few webservers myself, I knew that my rewrite directives themselves were correct, though the tech didn't believe me.

When I opened the ticket for this on Tuesday morning, there was a response at the end of the day saying that it was now enabled and should work. Of course, it didn't. On Wednesday by the end of office hours, they claimed once again that it should work and this time it did. A file had been added to the webserver's document root called web.config with the following contents:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<system.web>
		<httpHandlers>
			<add path="*.apehandler" verb="*" type="Helicon.Ape.Handler, Helicon.Ape, Version=3.0.0.64, Culture=neutral, PublicKeyToken=95bfbfd1a38437eb"                validate="false" />
		</httpHandlers>
		<httpModules>
			<add name="Helicon.Ape" type="Helicon.Ape.ApeModule, Helicon.Ape, Version=3.0.0.64, Culture=neutral, PublicKeyToken=95bfbfd1a38437eb" />
		</httpModules>
	</system.web>
</configuration>

I'm surprised that took nearly two workdays to figure out and fix, especially as it's supposedly enabled by default.

Conclusion

Normally it only takes a couple of hours to migrate a website to a new host by hand. In this case, it took 8 days of wall clock time, maybe four hours of which were the actual work needed to do the move. The remaining time was spent debating with the new hosting company's support staff. Now, I understand support staff get to deal with a lot of strange stuff from clueless customers, but when as a tech you do get a customer who knows what he's talking about (and actually tells you what's wrong and how it should be fixed), try to take note and act quickly rather than make a fool of yourself.

Tags: