It’s day 5 now and still no shipping label. I called HTC again and got a slightly different answer out of this rep. This one told me that my ticket had *just* been escalated and was on it’s way to the warehouse now. When I told him that the other agents had told me it had already gone up to the escalation team, he started backpedaling and said that although the ticket had been escalated it takes them time to look things over.

4 days? Ya, right.

So I asked for a supervisor. After waiting on hold (no music, just silence so you never know if you’ve been cut off), a ‘supervisor’ came on the line (supervisor is in quotes as, having working in a call centre before I have first hand experience pretending to be a co-workers supervisor – it’s a very common practice). I asked again what was going on, and she had yet another story. Apparently, HTC’s shipping system has been ‘down’ for ‘a few days’, and they have no ETA on things. She assured me, though, that (for real this time!) my ticket has been escalated, it should be at the attention of the warehouse, and I should have a label by Monday or Tuesday.

So, starting Monday afternoon (as nobody’s around to man the phones on weekends), I’ll continue with my daily phone campaign trying to find out when I can possibly expect to send back my wife’s brick…err…Nexus One.

Continued in Part 4.

Day four of the return process and no real progress to report. According to the details given to me by HTC on Monday (see part 1), I should have received a shipping label in my email the following day. When I didn’t, I called HTC support back. The agent I spoke with told me to check my spam folder (already done – nothing), and then said that she would ‘escalate the ticket’ and a label sent out right away. I thanked her, disconnected, and waited.

And waited.

End of day Wednesday rolled around and still no shipping label. I called HTC again and was told that it can take an extra day when things go through the escalation process and that I should expect it to show up in my email no later than Thursday.

Well, here we are on Thursday and still no label.

I called HTC yet again, and was told that yes, I should have received the shipping label today. The agent then proceeded to tell me that he update the ticket, ‘increase the priority’, and that I should keep waiting.

Really, HTC? I’m normally a pretty patient guy, but come on! I’ve heard of American customers getting new, replacement phones within days, but we Canadians need to go through this gong show of a return process and then re-purchase the device because otherwise we end up a refurbished phone instead of a new one. This is bad PR, people. It’s time to start spreading the news.

Find out what happens next in Part 3.

Her: Welcome to Tim Hortons, how can I help you?

Me: Hi. Can I please get an extra-large tea, with two milk and two sugar?

Her: <several seconds of silence> Ummm, so you want a decaf tea?

Me: Actually, I just want a regular tea.

Her: Okay. Do you want anything in it?

Me: Yes. Two milk, and two sugar.

Her: <several seconds more of silence>. So two cream, and one sweetener?

Me (trying not to sound frustrated): No. Two milk. <pause> And two sugar.

Her: Oh! Okay!

I’m surprised that I didn’t end up with a Double-Double instead, as has been known to happen before.

So Tuesday afternoon rolled around. I ran a manual backup of the Exchange server before IBM Dude came around and did a test restore to make sure everything was working, much like I should have done last time. As soon as he arrived, we powered down the server and swapped out the board. After everything was back in place, we crossed our fingers and pressed the power button.

-Click- WHIIIIIIIRRRRRRRRRRRRR

As the server powered on, we noticed two things. One was that the server sounded like a hurricane. With most servers, be they IBM or Dell, when you first turn them on all of the fans will spin up to full power, then settle down. In this case, the fans spun up, then stayed up. We could barely hear each other. The other thing we noticed, however, was an error message on-screen:

1604 Machine type mismatch detected

Neither of us panicked, though – we still had to flash the BIOS so we could put in the correct Machine Type and Serial numbers. The fans were starting to get annoying, though.

After the machine booted off the update CD, I plugged in the right numbers, double-and-triple-checking them, then let it do it’s thing. When it rebooted, the fans were as loud as ever, and, unfortunately, the error persisted.

1604 Machine type mismatch detected

Popping in to BIOS, I double-checked the Machine Type – it was set correctly. We both scratched our heads, and then noticed that the part number on the new board was different from the old one. In fact, after IBM Dude did a little searching, he found the new board was actually for an x3500, although it was supposedly a valid substitutable part. Regardless, and believing we’d found the problem, he ordered a new board of the correct part number and promised he’d be back Friday with the correct part. In the mean time, the server was still running, albeit a little slower and a lot louder, but at least now the power button was fixed and tape was no longer required.

As Friday afternoon came, I repeated my backup/test restore ritual, and once more IBM Dude arrived, box in hand. We doubled-checked the part number and confirmed that it was exactly the same as the original board that had failed. Downing the server yet again, he quickly swapped out the board and booted the system.

The hurricane sound persisted as all of the fans spun up to 100% and stayed there. As the POST messages came and went, we waited patiently for the RAID controller to initialize, and then were greeted with a familiar message:

1604 Machine type mismatch detected

Well crap. Once again we flashed the BIOS, but there was no change after putting in the correct serial number/machine type. Then an idea struck me.

Me: What about the BMC firmware?

Him: Well, I guess we can check that.

So we did! After downloading the latest version for the x3400 (1.29), we fired up the update utility and found something odd. The installed version was 1.41, which most definitely wasn’t listed on the website.

Me: Maybe that’s the problem – this version is probably for an x3500. They use the same board. Let’s just downgrade it.

Him: Err, I dunno. I’m gonna call my support guys and see what they have to say.

After a call in to IBM support, IBM Dude was told yep, downgrade the firmware as the 1.41 version was meant for the x3500 series. We did the deed, and surprise surprise, when the server rebooted the 1604 error was gone! However, the fans were still running in Tornado Mode.

Now for most Exchange administrators, there’s not a lot worse than when one of your storage groups isn’t mounting. Worse things would include the RAID array dying and the server catching fire (maybe one as a result of the other), or a user who decides that the server room doesn’t need air conditioning when nobody’s working in there and shuts it off over a long weekend.

Not that the last one has ever happened to anyone. *Cough*

Unfortunately, because I was an idiot and didn’t copy the error messages at the time (I was more worried about getting the server back up and running), I can only summarize what happened.

• Tried repeatedly to mount the database. As they say, if it doesn’t work the first time, it probably won’t work the seventh. Turns out, ‘they’ were right.
• Ran ‘chkdsk /r’ on the RAID array containing the transaction logs, and then on the array with the .edb – no love, still no mounting
• Tried every possible way to get eseutil /r to replay the transaction logs to the database, only to find that both were corrupt. Great.
• Tried to restore the last backup using Backup EXEC. It didn’t work.
• Admitted defeat and ran eseutil /p on the database.

Here’s the kicker: when running eseutil with the /p switch on a database that wasn’t shutdown cleanly or had the /r switch run on it first, all of the data in the transaction logs gets discarded. However, when they’re corrupt anyways, there’s really not a lot to lose.

When eseutil finally finished it’s repair after over an hour of grinding away, the database finally mounted. Heaving a sigh of relief I double-checked the tape and went home for night know I’d done all I could do. Surprisingly no one reported any missing emails the next morning, and I was able to grab a full backup of the server without issue.

When mid-afternoon rolled around, IBM Dude showed up with the ‘front diagnostic panel’, aka ‘the switch assembly’. We powered down the server, he ripped things apart, pulled out the old part, popped in the new one, and turned on the server.

Or at least, tried to turn it on.

–Click– *WHIRRRRRRRrrrrrr* –Click–

Fantastic. It looked like the first replacement switch assembly had the same problem. Ripping things apart again, IBM Dude swapped the replacement with the freshly ordered spare. Crossing our fingers, he tried the button again.

–Click– *WHIRRRRRRRrrrrrr* –Click–

Crap. At this point, he cut his losses and called for help. The suggestion? Replace the system board.

IBM Dude ordered the part, I booted the server again, once more relying on scotch tape to do it’s thing, and we made plans to have the board replaced the following Tuesday afternoon.

Will the gong show continue? Find out in Part 4.

Having determined that there was most definitely a hardware problem with the server, and now that it was back up and running (albeit with tape being the lynch-pin of the whole thing), I did a quick search for IBM’s support number and gave them a call. Surprisingly, there was no wait, and I was passed to an agent. Also not surprising, after spending ten minutes giving them information, they finally determined that my company had never contacted them for support before, and I had to give them my contact information again while they created an account. Once that was done, and the agent was armed with a brief description of my problem, I was transferred to Hardware Support.

The agent I spoke with there quickly agreed that this was a serious problem, and was slightly mortified that I was using tape to keep the server running, despite the fact I had little other choice. He also agreed with me that the problem was likely with the micro-switch in the power button and ordered a replacement for the whole ‘front panel diagnostic assembly’. That done, he informed me that a local technician would be contacting me shortly to confirm a time for a service call.

Apparently, despite the fact that replacing that particular part was nearly as easy as it gets, IBM, unlike Dell, insists on sending a tech. Whatever. I was just happy to get the problem fixed.

The next day, I received a call from the local IBM guy. He let me know that the part had come in and he’d drop by later that afternoon. I scheduled the downtime with my users and waited.

When he arrived, right on time, he didn’t have good news. Opening the box, we didn’t find the ‘front panel diagnostic assembly’, but instead the power button. Not the switch, mind you, but the actual solid-plastic-thing-that-your-finger-presses-which-in-turn-presses-the-switch. (If you are looking for where the power button is on an x3400, check out this post).

Chalking it up to a mistake at the call centre, IBM Dude ordered the correct part and told me he’d call back when it came in. I wasn’t really that concerned, as the tape was still holding on. Or so I thought.

The following morning (now on Day 3), I came in to find that the tape had betrayed me and set the button free during the night, taking the server down again. After a little fighting, as now the server didn’t want to power up on the first press – I had to fight with it a little to get it actually turn on, I re-taped the switch with electrical tape, and crossed my fingers that the Exchange databases and logs were intact. Fortunately, they were.

Not long after, IBM Dude called again to say the right part was in. However, after ‘testing’ the switch a few times, the though he heard a *snap* sound and the button wasn’t quite pressing right anymore. As a result, he said that he’d order two more switches and let me know when they’d come in. Once again, I wasn’t that worried. Sure, it was Thursday, but surely the replacements would be in by the next day, and the whole thing would be fixed by then. I left early for the day, and was nearly home, getting ready to relax, when my cell rang. The Exchange server was down again.

Swearing under my breath as I wandered back in to my office, I found that the electrical tape had let go – apparently it doesn’t stick to a metal case anywhere near as well as scotch tape. I fought with the switch a little more before getting it powered on again, and this time I secured the button with enough tape to silence a large choir. After confirming it booted to Windows, I was just about to leave again when a staff member came by and said they still couldn’t get in to Outlook.

Consoling in to the server, I loaded up the Exchange Management Console and had a look around. That’s when I discovered the the First Storage Group wasn’t mounting.

Shit.

(Continued in Part 3).

Sure enough, though, I didn’t even have a chance to sit down at my desk before my phone was ringing with complaints that Outlook was throwing errors saying it couldn’t connect to the Exchange server. I turned on the LCD for my server monitor, and sure enough, the server wasn’t responding. Grabbing my keys, I trundled up to the server room, still unconcerned as the rest of the staff would be heading for a meeting and as such, I’d have an hour to get things sorted out.

When I got to the sever room, I was greeted by the whirring of cooling fan, however the sound was a little off – the pitch was different than normal, and not quite as loud. Glancing at the server rack, my suspicions were confirmed: the server was off.

No problem, I figured, pulling off the front panel of the IBM x3400 to uncover the power button, just need to flick it back on. I pressed the button (note: if you’re here looking for how to find the power button on an x3400, check out this post).

–Click– *WHIRRRRRRRrrrrrr* –Click–

The server had turned off as quickly as it came on. I pressed the power button again, with much the same result. Crap. Maybe this wouldn’t be so fast after all.

Read more after the break.

As the server room was very cramped, I grabbed a cart and hauled the server down the workbench beside my desk. After hooking everything up, I tried again, once more with it spinning up the fans and then powering off immediately.

On a hunch, I pressed the power button more time, but this time held it in.

–Click– *WHHHHIIIRRRRRRRRRRRR*

Sure enough, the server stayed on, and the P.O.S.T. screen came up. Letting go of the power button, the server –Click–’ed off.

Dammit.

Grabbing some scotch tape, I held the power button in again, secured it to the case, and let go. It held. At this point, the uptime and reliability of my server relied on clear plastic sticky tape. Perfect. Grabbing the phone, I called IBM support. And that’s where the real gong show started….

(Continued in Part 2)