Fast-forward to last week. With my wifes birthday coming up, I thought it would be nice to surprise her with a trip to the local Telus dealer (her carrier) for a shiny new phone. After looking at their offerings (the only two real considerations were the Motorola Milestone and the HTC Hero), she decided that she’d come down with a case of Phone Envy and wanted a Nexus One as well.

So, no big deal – we ordered the phone and waited. Unfortunately, due to the stat holiday on July 1st, the phone didn’t arrive on Friday like expected and instead came in yesterday (the following Monday). I left the phone plugged in to charge, and when the light turned green to say all was well I fired it up.

And waited.

And waited some more.

The Boot Screen on my other, unlocked (and functional) Nexus One

After about five minutes of staring at the above picture (minus the lock on the bottom, as I hadn’t gotten around to doing that yet), I got the feeling something was wrong.

I powered off the phone by pulling out the battery, then booted in to HBOOT mode (hold down the VOLUME DOWN key while pressing Power). That worked fine, so I tried to enter recovery. Unfortunately, I was met with the same screen. I tried several more times for both normal mode and recovery without success. Then I gave up and called Google.

At this point, I’d like to point out that I’m fairly certain it’s just a software error. Unfortunately, the only way I could really do anything about that would be to Unlock the phone, which voids the warranty. As such, the call to support was warranted.

I called the number listed on the Google Phone Contact Support page (1-888-48-NEXUS) and after a few quick IVR choices I was greeted nearly immediately by a tech. He asked about the problem, what I had tried, and immediately agreed the phone was D0A and would need to be replaced. That’s when things started to go downhill.

The problem is that I live in Canada, and HTC has a different policy for international orders than their standard ‘send you a replacement and then send the old one back’ policy for domestic ones. If you live in Canada and get a DoA phone, you have two options:

1. Send it back via their normal repair process and get a refurbished phone (no option for a new one)
2. Initiate a ‘Buyers Remorse’ return, get a refund for the device, and then order a new one.

The choice was fairly obvious – I ordered my wife a new phone, so that’s what she was going to get. After gathering a little more information, the Google tech transfered me over to HTC support to process the return order.

As the new tech was going over the specifics, a thought occurred:

Me: Okay, so I do the return, and within 14 business days of you receiving the device and okay’ing everything, you credit the purchase price back, correct?

Tech: Yes, that’s correct.

Me: So what about the duty charge that I had to pay CoD?

Tech: Oh, that. Hang on. –Pause– Okay, I checked on that, and what happens is you’ll get some paperwork from us once the refund has been processed. Once you have that paperwork, contact Customs and they’ll be able to issue a refund.

So at this point, I may have to wait three weeks for credit from HTC (although apparently it usually doesn’t take that long), and then I can file a claim to get my ~$70CAD that I paid for taxes back.

All was said and done, and I prompted received a confirmation email repeating what the agent said. However, as of posting I still haven’t received a FedEx shipping label in my email as promised. I called HTC back, and they escalated the ticket and said they’d resend the label, so now I’m waiting on that. Once I’ve got the label in and the phone sent away, I’ll order another N1 and see what happens.

The pain begins in Part 2!

Click ‘Continue Reading’ for the wall-of-text!

To start, we need to get a little background out of the way. When you buy a retail copy of Windows, you get a Retail disk with a Retail key. You put the key in and life is usually good – Windows activates online and you’re done. Occasionally, maybe due to a hardware change, Windows refuses to activate online and instead makes you call in. You follow the steps, punch in the numbers, and you’re done. Pretty simple.

However, when it comes to a computer with Windows pre-installed by a major manufacturer, things are a little different.

You see, to make things easier for the average user, Microsoft added a feature called ‘System Locked Preinstallation‘, or ‘SLP’. To summarize the linked Wikipedia article, Microsoft provides Manufactures with a set of OEM Product Keys. The manufacturer then gives you these keys on the COA affixed to your computer and an installation disk with a generic Product Key pre-entered. When you do the install, Windows checks the Product Key, and if it finds an OEM SLP key, it looks for a Digital Certificate that the manufacturer embedded in the BIOS. When it finds the cert, it ‘activates’ without going online.

The problem I have is that I rarely (read: never) use the manufacturer-provided Windows installation disk, mainly to avoid the crap they put on them (‘free’ trials of Norton Antivirus, crapware, etc…). As such, with Windows XP I’d use a generic System Builder OEM disk, or with Windows Vista/7, just a generic Retail disk (with Windows Vista, all the SKUs are on one disk, and Windows automatically detects which edition to install based on your product key. Windows 7 disks also have almost all SKUs present, but you need to edit/delete a file first to access that feature).

When using a generic disk, the SLP process no longer applies, and you have to put in the Product Key that is on the COA attached to your computer. Although the key will enter fine, it won’t activate online. This is because although it’s a valid OEM key, it’s technically never supposed to be used, as the OEM-provided installation disk will use a different, generic key. As such, you need to call in to Microsoft’s Product Activation line in order to activate the key on your COA.

Now comes the fun part. I recently received a bunch of new Dell systems and went through the telephone activation process as usual. Everything worked perfectly. Then today, I had to re-install Windows Vista Business on a year-old laptop. After the installation, I put in the product key and went to the telephone activation line. After punching in the verification numbers, I heard this:

We’re sorry, but we were unable to validate your installation ID. Please hold while we transfer you to a representative.

So, I waited on hold for a few minutes was greeted with the following:

Agent: Thank you for contacting the Microsoft Telephone Activation department. Can I start with the first block of numbers on the activation screen?

Me: Yes, they’re XXXXXX.

Agent: Okay, is the second block YYYYYY?

Me: Yes.

Agent: Perfect. And what product are you trying to activate?

Me: Windows Vista Business Edition.

Agent: Okay. Please enter the following numbers in to the activation screen. <insert numbers here>

Me: Okay, it activated.

Agent: Thank you!

What the hell? He used the numbers I entered in to the automated part of the phone system, so I didn’t type them wrong, and he didn’t ask me the usual questions (“How many computers is this installed on, et al…”), so why did the automated activation fail? Have they changed the process? Am I going to have to go through this every time? Should I just say ‘screw it’ and only order VLK product keys instead and tell the OEMs to not bother with Windows anymore?

Find out the answer to these questions and more…well, probably never. [/end rant]

Any method of RAID (other than RAID0) is good for protecting against failing hard drives. However, it is by no means a backup solution, as it doesn’t save your data from things like accidental deletion, file system corruption, users, and malicious software. If you’re looking for a backup solution, investigate external hard drives, SAN/NAS devices, tape backup units, etc….

Please. For the sake of your data, get a real backup solution.

10:04:03 AM System You are now being connected to an agent. Thank you for using Dell Chat
10:04:03 AM System Connected with ***_Oliver_******

10:04:08 AM ***_Oliver_******: Hi, thank you for contacting DELL Canada support, my name is Oliver, how may I help you?

10:05:01 AM Laslow: I have a system that is unable to POST. If I remove the video card and use the onboard video, it POSTs fine. If I take the video card from an identical unit and try that, it also works fine. It appears that the video card is defective, and I need a replacement sent out.

10:07:13 AM ***_Oliver_******: I’ll do my best to assist you with this one.
10:07:48 AM ***_Oliver_******: I could see here that system is an ***** and warranty is active until *****.

10:07:55 AM Laslow: That is correct.

10:08:58 AM ***_Oliver_******: I’ll replace this.
10:09:13 AM ***_Oliver_******: Would you like me to send an onsite tech ?

10:09:21 AM Laslow: No thank you – the part will be fine.

10:09:33 AM ***_Oliver_******: May I have the shipping address?

10:09:58 AM Laslow: ***** ***** ***** *****

10:11:23 AM ***_Oliver_******: By the way, may I know who’s the registered owner of the system?

10:12:46 AM Laslow: ***** ***** ***** *****

10:13:28 AM ***_Oliver_******: Thanks. I’ll go ahead and process the case and dispatch. This would take me about 5 minutes. Hold on.

10:13:34 AM Laslow: Thank you.

10:13:39 AM ***_Oliver_******: Would you like me to send the reference numbers thru email instead?

10:13:50 AM Laslow: That would be greatly appreciated.

10:14:49 AM ***_Oliver_******: Please wait for the part next business day. Return to us the old part not later than ten days once the replacement part is received. You could use the same box that came in with it.
10:14:54 AM ***_Oliver_******: Anything else I can help you with?

10:15:09 AM Laslow: No, that is fine. Thank you very much! You have been very helpfull!

10:15:19 AM ***_Oliver_******: Would you like me to contact you to followup on this?

10:15:36 AM Laslow: No, that will not be required.

10:17:39 AM ***_Oliver_******: Ok. Thank you! Thank you for choosing DELL, have a good day!

10:17:51 AM Laslow: Thanks again!

As the system was a Dell, it had a set of Diagnostic LEDs on the front to help diagnose the problem. Unfortunately, looking up the Optiplex 745 manual, it didn’t list the particular code the lights were showing: 1 3 4.

Looking through the listed codes, most of the ones related to 1 or 3 4 were for either faulty RAM or a bad memory controller, so I started there. I tried removing one stick, then replacing it and removing the other, using a completely different stick, and then none at all. Each time I was met with the same diagnostic code, 1 3 4.

Convinced I had the problem pin-pointed, I tried to get on to Dell’s Live Chat Hardware Support to get a replacement motherboard as the system was still covered by Next Business Day parts. The site, unfortunately, was down, so I was forced to call Dell instead.

Now, there’s a reason that I typically use Live Chat instead of the phone. The main reason is on Live Chat, I can quickly bang-out the troubleshooting I’ve already done, and 9 times out of 10, the tech will just send me the part. Phone support is rarely that easy, and this time was no exception.

The wait time was average (maybe about 5 minutes), and once I got the agent on, I quickly rattled off my troubleshooting and told him the diagnostic code. He put me on hold for a few minutes and when he came back, told me that the diagnostic code 1 3 4 on an Optiplex 745 means the video card isn’t working. Figuring it would be easy enough to test, I pulled out the PCI-E video card and switch to the onboard video. Sure enough, it worked.

At this point, I figured we’d be done. I didn’t have another PCI-E card to make sure it wasn’t the slot, but I supposed out loud that if I got the replacement card and it didn’t work, I could always send it back, make another call, and get a motherboard sent out. The ‘tech’ had another idea, though.

Him: There’s actually one more troubleshooting step I’d like to try.

Me: Okay, what’s that?

Him: Well, leave the video card out, boot in to Windows, go to Device Manager and remove the device. Then turn the computer off, put the video card back in, and see if it work.

Him: I just want to see if the driver is the problem.

Me: …what?

At this point, I nearly hung up the phone. Instead, I figured a little education would be in order.

Me: Okay, I think you might be a little confused. The problem is that, with the video card in, the system won’t even POST properly. I fail to see how a Windows driver could possibly be the problem, because at this stage in the boot process, the computer has no clue what operating system is on the hard drive. It hasn’t even looked to see if there’s a hard drive there. How can a driver possibly be the problem.

Him: Sir, it’s just part of my troubleshooting process.

Me: Okay, I’ll humor you. I’m going to pretend to boot my computer.

Him: No, I need you to actually–

Me (interrupting): Now I’m going to pretend to open Device Manager. Oh, would you look at that! Because the video card isn’t present, it’s not showing in the list.*

Him: …oh. Well, I guess we can send you a replacement video card….

Me: I thought so.

This morning, I received a call back from Dell asking me if I’d received the video card. I said no, and he told me that the card had been delivered and signed for. Figuring that maybe Purolator had delivered it to the wrong office, I put the agent on hold and checked around, but nobody had received anything today. Going back to the agent, I asked who signed for it, at which point he informed me that it hadn’t been delivered yet and would likely arrive this afternoon. Gee, thanks for that.

*Note: I’m aware that you can make Device Manager show non-present device by following this KB article, however that would have only confused the poor agent and wouldn’t have made a difference anyway.

So Tuesday afternoon rolled around. I ran a manual backup of the Exchange server before IBM Dude came around and did a test restore to make sure everything was working, much like I should have done last time. As soon as he arrived, we powered down the server and swapped out the board. After everything was back in place, we crossed our fingers and pressed the power button.

-Click- WHIIIIIIIRRRRRRRRRRRRR

As the server powered on, we noticed two things. One was that the server sounded like a hurricane. With most servers, be they IBM or Dell, when you first turn them on all of the fans will spin up to full power, then settle down. In this case, the fans spun up, then stayed up. We could barely hear each other. The other thing we noticed, however, was an error message on-screen:

1604 Machine type mismatch detected

Neither of us panicked, though – we still had to flash the BIOS so we could put in the correct Machine Type and Serial numbers. The fans were starting to get annoying, though.

After the machine booted off the update CD, I plugged in the right numbers, double-and-triple-checking them, then let it do it’s thing. When it rebooted, the fans were as loud as ever, and, unfortunately, the error persisted.

1604 Machine type mismatch detected

Popping in to BIOS, I double-checked the Machine Type – it was set correctly. We both scratched our heads, and then noticed that the part number on the new board was different from the old one. In fact, after IBM Dude did a little searching, he found the new board was actually for an x3500, although it was supposedly a valid substitutable part. Regardless, and believing we’d found the problem, he ordered a new board of the correct part number and promised he’d be back Friday with the correct part. In the mean time, the server was still running, albeit a little slower and a lot louder, but at least now the power button was fixed and tape was no longer required.

As Friday afternoon came, I repeated my backup/test restore ritual, and once more IBM Dude arrived, box in hand. We doubled-checked the part number and confirmed that it was exactly the same as the original board that had failed. Downing the server yet again, he quickly swapped out the board and booted the system.

The hurricane sound persisted as all of the fans spun up to 100% and stayed there. As the POST messages came and went, we waited patiently for the RAID controller to initialize, and then were greeted with a familiar message:

1604 Machine type mismatch detected

Well crap. Once again we flashed the BIOS, but there was no change after putting in the correct serial number/machine type. Then an idea struck me.

Me: What about the BMC firmware?

Him: Well, I guess we can check that.

So we did! After downloading the latest version for the x3400 (1.29), we fired up the update utility and found something odd. The installed version was 1.41, which most definitely wasn’t listed on the website.

Me: Maybe that’s the problem – this version is probably for an x3500. They use the same board. Let’s just downgrade it.

Him: Err, I dunno. I’m gonna call my support guys and see what they have to say.

After a call in to IBM support, IBM Dude was told yep, downgrade the firmware as the 1.41 version was meant for the x3500 series. We did the deed, and surprise surprise, when the server rebooted the 1604 error was gone! However, the fans were still running in Tornado Mode.

Now for most Exchange administrators, there’s not a lot worse than when one of your storage groups isn’t mounting. Worse things would include the RAID array dying and the server catching fire (maybe one as a result of the other), or a user who decides that the server room doesn’t need air conditioning when nobody’s working in there and shuts it off over a long weekend.

Not that the last one has ever happened to anyone. *Cough*

Unfortunately, because I was an idiot and didn’t copy the error messages at the time (I was more worried about getting the server back up and running), I can only summarize what happened.

• Tried repeatedly to mount the database. As they say, if it doesn’t work the first time, it probably won’t work the seventh. Turns out, ‘they’ were right.
• Ran ‘chkdsk /r’ on the RAID array containing the transaction logs, and then on the array with the .edb – no love, still no mounting
• Tried every possible way to get eseutil /r to replay the transaction logs to the database, only to find that both were corrupt. Great.
• Tried to restore the last backup using Backup EXEC. It didn’t work.
• Admitted defeat and ran eseutil /p on the database.

Here’s the kicker: when running eseutil with the /p switch on a database that wasn’t shutdown cleanly or had the /r switch run on it first, all of the data in the transaction logs gets discarded. However, when they’re corrupt anyways, there’s really not a lot to lose.

When eseutil finally finished it’s repair after over an hour of grinding away, the database finally mounted. Heaving a sigh of relief I double-checked the tape and went home for night know I’d done all I could do. Surprisingly no one reported any missing emails the next morning, and I was able to grab a full backup of the server without issue.

When mid-afternoon rolled around, IBM Dude showed up with the ‘front diagnostic panel’, aka ‘the switch assembly’. We powered down the server, he ripped things apart, pulled out the old part, popped in the new one, and turned on the server.

Or at least, tried to turn it on.

–Click– *WHIRRRRRRRrrrrrr* –Click–

Fantastic. It looked like the first replacement switch assembly had the same problem. Ripping things apart again, IBM Dude swapped the replacement with the freshly ordered spare. Crossing our fingers, he tried the button again.

–Click– *WHIRRRRRRRrrrrrr* –Click–

Crap. At this point, he cut his losses and called for help. The suggestion? Replace the system board.

IBM Dude ordered the part, I booted the server again, once more relying on scotch tape to do it’s thing, and we made plans to have the board replaced the following Tuesday afternoon.

Will the gong show continue? Find out in Part 4.

Environment: Backup EXEC 12.5 SP2 (fully updated) running on Windows Server 2008 SP2, backing up, among other things, Exchange Server 2007 SP2 x64, running on Windows Server 2003 x64.

Setup: Due to horrible issues with Backup EXEC’s (referred to as ‘BE’ from now on) handling of GRT (Granular Restore Technology) with tape backups, BE runs a B2D (Backup to Disk) job to backup the Exchange mailboxes, and then a second B2T (Backup to Tape) job does the standard backup, plus grabs the B2D directory.

Problem: The B2D job fails intermittently with the following error:

Cannot extract mailbox messages from the Exchange backup. V-79-57344-759. -1213 The database page size does not match the engine on the backup to disk job.

Troubleshooting: According to this article, Hotfix 319699 is supposed to resolve the issue, in most cases. According to the article, though, it only applies to 12.0 and Service Packs, 12.5, and 12.5 SP1. The issue is supposed to be resolved in SP2, which I’m currently running.

After spending an hour on the phone with Symantec support, as the article says to call them if the Hotfix doesn’t resolve the issue (it didn’t for me), I was basically told that the version of BERestore.exe in the C:Program FilesSymantecBackup ExecRAWS directory was older than the version on the BE Server (12.5.2213.101 compared to 12.5.2213.144).

Resolution: So far, the fix seems to be to manually copy the C:Program FilesSymantecBackup ExecAgentsRAWSX64Updates directory to the Exchange server, then sort by date and apply RAWSx642213RSP2.msp, plus every update newer than that. Once the updates are complete (it took me about 10 minutes to run all 8 applicable patches), reboot the server.

After doing so, I ran the B2D job and it completed, however that isn’t necessarily proof that this worked – as the problem is intermittent, it’ll take another week of scheduled jobs to tell for sure whether that resolved the problem.

For the time being, my fingers are crossed.

Having determined that there was most definitely a hardware problem with the server, and now that it was back up and running (albeit with tape being the lynch-pin of the whole thing), I did a quick search for IBM’s support number and gave them a call. Surprisingly, there was no wait, and I was passed to an agent. Also not surprising, after spending ten minutes giving them information, they finally determined that my company had never contacted them for support before, and I had to give them my contact information again while they created an account. Once that was done, and the agent was armed with a brief description of my problem, I was transferred to Hardware Support.

The agent I spoke with there quickly agreed that this was a serious problem, and was slightly mortified that I was using tape to keep the server running, despite the fact I had little other choice. He also agreed with me that the problem was likely with the micro-switch in the power button and ordered a replacement for the whole ‘front panel diagnostic assembly’. That done, he informed me that a local technician would be contacting me shortly to confirm a time for a service call.

Apparently, despite the fact that replacing that particular part was nearly as easy as it gets, IBM, unlike Dell, insists on sending a tech. Whatever. I was just happy to get the problem fixed.

The next day, I received a call from the local IBM guy. He let me know that the part had come in and he’d drop by later that afternoon. I scheduled the downtime with my users and waited.

When he arrived, right on time, he didn’t have good news. Opening the box, we didn’t find the ‘front panel diagnostic assembly’, but instead the power button. Not the switch, mind you, but the actual solid-plastic-thing-that-your-finger-presses-which-in-turn-presses-the-switch. (If you are looking for where the power button is on an x3400, check out this post).

Chalking it up to a mistake at the call centre, IBM Dude ordered the correct part and told me he’d call back when it came in. I wasn’t really that concerned, as the tape was still holding on. Or so I thought.

The following morning (now on Day 3), I came in to find that the tape had betrayed me and set the button free during the night, taking the server down again. After a little fighting, as now the server didn’t want to power up on the first press – I had to fight with it a little to get it actually turn on, I re-taped the switch with electrical tape, and crossed my fingers that the Exchange databases and logs were intact. Fortunately, they were.

Not long after, IBM Dude called again to say the right part was in. However, after ‘testing’ the switch a few times, the though he heard a *snap* sound and the button wasn’t quite pressing right anymore. As a result, he said that he’d order two more switches and let me know when they’d come in. Once again, I wasn’t that worried. Sure, it was Thursday, but surely the replacements would be in by the next day, and the whole thing would be fixed by then. I left early for the day, and was nearly home, getting ready to relax, when my cell rang. The Exchange server was down again.

Swearing under my breath as I wandered back in to my office, I found that the electrical tape had let go – apparently it doesn’t stick to a metal case anywhere near as well as scotch tape. I fought with the switch a little more before getting it powered on again, and this time I secured the button with enough tape to silence a large choir. After confirming it booted to Windows, I was just about to leave again when a staff member came by and said they still couldn’t get in to Outlook.

Consoling in to the server, I loaded up the Exchange Management Console and had a look around. That’s when I discovered the the First Storage Group wasn’t mounting.

Shit.

(Continued in Part 3).