PFDAVAdmin and the Case of “Could Not Expand”

17. June 2011 · Write a comment · Categories: Microsoft, Software, Troubleshooting, Windows · Tags: , , ,

There are a number of articles out there about how to bulk-update permissions on calendars in Microsoft Exchange, most of them pointing to the PFDAVAdmin tool. The problem, though, is that you have to read the requirements for it very carefully. Case in point:

PFDAVAdmin "could not expand"I ran in to this when trying to run the tool from my Server 2003 x64-based Exchange 2007 server. It happened again when I tried to run it from a Server 2008 x64 box, and from my Windows 7 x64 workstation.

As it turns out, PFDAVAdmin requires .Net Framework 1.1 to be installed. It isn’t recommended to install that directly on to your Exchange Server as it can cause issues with .Net 2.0, so I simply installed it on my Win7 x64 box, ignored the Compatibility Warning, and that was it – PFDAVAdmin worked perfectly.

Gong Show: The Tail of an IBM x3400 Server (Part 4)

30. September 2009 · 3 comments · Categories: Hardware, Servers, Support · Tags: , , ,

If you came here looking for information on where to find the power button on an IBM x3400 or x3500, check this post instead.
(Continued from Part 3)

So Tuesday afternoon rolled around. I ran a manual backup of the Exchange server before IBM Dude came around and did a test restore to make sure everything was working, much like I should have done last time. As soon as he arrived, we powered down the server and swapped out the board. After everything was back in place, we crossed our fingers and pressed the power button.

-Click- WHIIIIIIIRRRRRRRRRRRRR

As the server powered on, we noticed two things. One was that the server sounded like a hurricane. With most servers, be they IBM or Dell, when you first turn them on all of the fans will spin up to full power, then settle down. In this case, the fans spun up, then stayed up. We could barely hear each other. The other thing we noticed, however, was an error message on-screen:

1604 Machine type mismatch detected

Neither of us panicked, though – we still had to flash the BIOS so we could put in the correct Machine Type and Serial numbers. The fans were starting to get annoying, though.

After the machine booted off the update CD, I plugged in the right numbers, double-and-triple-checking them, then let it do it’s thing. When it rebooted, the fans were as loud as ever, and, unfortunately, the error persisted.

1604 Machine type mismatch detected

Popping in to BIOS, I double-checked the Machine Type – it was set correctly. We both scratched our heads, and then noticed that the part number on the new board was different from the old one. In fact, after IBM Dude did a little searching, he found the new board was actually for an x3500, although it was supposedly a valid substitutable part. Regardless, and believing we’d found the problem, he ordered a new board of the correct part number and promised he’d be back Friday with the correct part. In the mean time, the server was still running, albeit a little slower and a lot louder, but at least now the power button was fixed and tape was no longer required.

More »

More Exchange Madness!

29. September 2009 · Write a comment · Categories: howto, Microsoft, Servers, Software, Troubleshooting · Tags: , , ,

I’ll just sum this one up, as it’s pretty boring, but there are some important details.

1) Computer account for a Domain Controller/Global Catalogue/Exchange server (virtualized in Hyper-V) becomes corrupt, including the underlying metadata.

2) That server cannot ‘see’ the domain, with numerous errors from both Exchange and the Event Viewer stating that it cannot replicate due to DNS problems.

3) Rolling the .vhd back to a previous week results in the same issue.

4) When attempting to demote/dejoin/join/promote the server from/to the domain, the computer account is deleted, but not the metadata, and the server cannot be joined to the domain again.

Solution? Backup the Mailbox Databases from the First and Second Storage Groups, as well as their transaction logs, then create a new .vhd, reinstall the OS, join it to the domain, add the newly created computer account to the ‘Exchange Server’ and ‘Exchange Install Domain Servers’ groups, install Exchange using “setup.com /m:recoverserver” (make sure that you’ve manually installed the prerequisites, such as IIS+IIS 6 Management Console, etc… before doing this), then copy the Mailbox databases back to the default install location. After that, correct the permissions on the Mailbox folder if needed (simply inherit the permissions from the parent object) and reboot the server. When it finishes booting, open the Exchange Management Console and mount the Storage Groups (note: you may have to open the properties on both groups and uncheck the option that prevents Exchange from automatically mounting the databases on boot).

Simple, right?

Gong Show: The Tail of an IBM x3400 Server (Part 3)

14. September 2009 · Write a comment · Categories: Hardware, Servers, Support, Troubleshooting · Tags: , , ,

(Continued from Part 2)

Now for most Exchange administrators, there’s not a lot worse than when one of your storage groups isn’t mounting. Worse things would include the RAID array dying and the server catching fire (maybe one as a result of the other), or a user who decides that the server room doesn’t need air conditioning when nobody’s working in there and shuts it off over a long weekend.

Not that the last one has ever happened to anyone. *Cough*

Unfortunately, because I was an idiot and didn’t copy the error messages at the time (I was more worried about getting the server back up and running), I can only summarize what happened.

  • Tried repeatedly to mount the database. As they say, if it doesn’t work the first time, it probably won’t work the seventh. Turns out, ‘they’ were right.
  • Ran ‘chkdsk /r’ on the RAID array containing the transaction logs, and then on the array with the .edb – no love, still no mounting
  • Tried every possible way to get eseutil /r to replay the transaction logs to the database, only to find that both were corrupt. Great.
  • Tried to restore the last backup using Backup EXEC. It didn’t work.
  • Admitted defeat and ran eseutil /p on the database.

Here’s the kicker: when running eseutil with the /p switch on a database that wasn’t shutdown cleanly or had the /r switch run on it first, all of the data in the transaction logs gets discarded. However, when they’re corrupt anyways, there’s really not a lot to lose.

When eseutil finally finished it’s repair after over an hour of grinding away, the database finally mounted. Heaving a sigh of relief I double-checked the tape and went home for night know I’d done all I could do. Surprisingly no one reported any missing emails the next morning, and I was able to grab a full backup of the server without issue.

When mid-afternoon rolled around, IBM Dude showed up with the ‘front diagnostic panel’, aka ‘the switch assembly’. We powered down the server, he ripped things apart, pulled out the old part, popped in the new one, and turned on the server.

Or at least, tried to turn it on.

–Click– *WHIRRRRRRRrrrrrr* –Click–

Fantastic. It looked like the first replacement switch assembly had the same problem. Ripping things apart again, IBM Dude swapped the replacement with the freshly ordered spare. Crossing our fingers, he tried the button again.

–Click– *WHIRRRRRRRrrrrrr* –Click–

Crap. At this point, he cut his losses and called for help. The suggestion? Replace the system board.

IBM Dude ordered the part, I booted the server again, once more relying on scotch tape to do it’s thing, and we made plans to have the board replaced the following Tuesday afternoon.

Will the gong show continue? Find out in Part 4.

Gong Show: The Tail of an IBM x3400 Server (Part 2)

09. September 2009 · Write a comment · Categories: "It's a Feature", Hardware, Servers, Support, Troubleshooting · Tags: , , , ,

(Continued from Part 1)

Having determined that there was most definitely a hardware problem with the server, and now that it was back up and running (albeit with tape being the lynch-pin of the whole thing), I did a quick search for IBM’s support number and gave them a call. Surprisingly, there was no wait, and I was passed to an agent. Also not surprising, after spending ten minutes giving them information, they finally determined that my company had never contacted them for support before, and I had to give them my contact information again while they created an account. Once that was done, and the agent was armed with a brief description of my problem, I was transferred to Hardware Support.

The agent I spoke with there quickly agreed that this was a serious problem, and was slightly mortified that I was using tape to keep the server running, despite the fact I had little other choice. He also agreed with me that the problem was likely with the micro-switch in the power button and ordered a replacement for the whole ‘front panel diagnostic assembly’. That done, he informed me that a local technician would be contacting me shortly to confirm a time for a service call.

Apparently, despite the fact that replacing that particular part was nearly as easy as it gets, IBM, unlike Dell, insists on sending a tech. Whatever. I was just happy to get the problem fixed.

More »

Gong Show: The Tail of an IBM x3400 Server (Part 1)

07. September 2009 · 1 comment · Categories: "It's a Feature", Hardware, Microsoft, Servers, Support, Troubleshooting · Tags: , , ,

It was Tuesday morning, just after the August Long Weekend. I’d spent much of the weekend doing as little as possible, but had a feeling of great dread as I walked in to my office. Maybe it was because things had been running smoothly for weeks without a major problem. Maybe it was because there always seems to be extra work to do after long weekends. Or maybe, just maybe, it was because my iPhone hadn’t been able to connect to our Exchange 2007 server since Saturday morning. I hadn’t been horribly concerned – OWA support was only in testing for our organization and I was the only user.

Sure enough, though, I didn’t even have a chance to sit down at my desk before my phone was ringing with complaints that Outlook was throwing errors saying it couldn’t connect to the Exchange server. I turned on the LCD for my server monitor, and sure enough, the server wasn’t responding. Grabbing my keys, I trundled up to the server room, still unconcerned as the rest of the staff would be heading for a meeting and as such, I’d have an hour to get things sorted out.

When I got to the sever room, I was greeted by the whirring of cooling fan, however the sound was a little off – the pitch was different than normal, and not quite as loud. Glancing at the server rack, my suspicions were confirmed: the server was off.

No problem, I figured, pulling off the front panel of the IBM x3400 to uncover the power button, just need to flick it back on. I pressed the button (note: if you’re here looking for how to find the power button on an x3400, check out this post).

–Click– *WHIRRRRRRRrrrrrr* –Click–

The server had turned off as quickly as it came on. I pressed the power button again, with much the same result. Crap. Maybe this wouldn’t be so fast after all.

Read more after the break.

More »

Server Complete

06. April 2009 · Write a comment · Categories: Hardware, Servers, Software · Tags: , , ,

So my idea of live blogging the setup and configuration of the new server fizzled. Too much to do and not enough time to do it. Between recovering critical user data from a failed hard drive, writing batch scripts to grab PST files and replace Word templates, and importing a whole mess of other stuff there just wasn’t enough weekend.

All in all, it went well. The physical server was setup with Hyper-V and the Symantec Endpoint management apps. The Terminal Server VM was given 8GB RAM and a 125GB VHD, while the Exchange Server (which, until our ISP gives us the go-ahead to actually run a mail server, currently only deals with Mailboxes and Public Folders) was given 6GB of RAM and a 125GB VHD of it’s own. That leaves 2GB of RAM for the physical server.

This morning was go-live, and by mid morning all 25 employees were logged in and active without any serious issues. Best of all, even with all of them in Outlook and running a myriad of apps, everything ran smooth as silk.

Once I’ve made sure everything is working fine, I’ll post a tutorial for backing up VHDs with Windows Server Backup and VSS.