« Posts under Servers

ADMT: What I Learned

March 29, 2011 / posted in howto, Microsoft, Networking, Servers, Troubleshooting, Windows / No Comments

Years ago, long before I started working at my current job, management launched a new contract in a office building just across the street. At the time, wireless network connections were still in their infancy and not to be trusted, so the new office was set up with a pair of servers, a nice new Active Directory Forest and Domain (DomainB), and a VPN to access resources on the primary network, DomainA.

Fast forward to three years ago, just before I was hired. The then-sysadmin was getting flak for the VPN being slow, so he installed a pair of wireless routers on the roofs of the buildings and linked the two networks. However, instead of getting rid of DomainB, he simply left it in place.

Fast forward to now. Due to cost issues, the contact in the remote office was physically moved to our main building. As such, their network equipment and servers came with them, which created cramped quarters in an already cramped space. As such, I set about doing what should have been done years ago – migrating users from DomainB to DomainA.

There was a group of client computers that needed to go through a round of updates anyways, so those were simply re-imaged and joined to a separate, restricted network (DomainC) used for our clients only (this had been another pet peeve of mine – due to costs, the clients in that office were put on the same network and although they had their permissions restricted, it was still a concern in my mind). The main problem, though, was the staff workstations. Not only were they setup on DomainB, put PrimaryDC.DomainB was also an Exchange 2003 server, and TertiaryDC.DomainA was our primary mail server running Exchange 2007. The first step was to manually export the mail for the twelve staff members and create their DomainA accounts, and then get them setup on the DomainA Exchange server. Once that was up and running, the Exchange 2003 install was shutdown. Although it took a while to manually transfer the mail by exporting to .PST files and then importing it again, it was the cleanest way to do the move (and also encouraged users to clean out their mailboxes).

The last step was to actually get the users logging in to DomainA rather than DomainB. That’s where ADMT (Active Directory Migration Tool) comes in.

ADMT comes in a few ‘current’ versions. 3.0 if the server it’s running on is Server 2003, 3.1 if it’s Server 2008, and 3.2 if it’s Server 2008 R2. The source domain (B) was running on Server 2003 boxes, but the target domain (A) was running mostly on Server 2008 boxes, so I installed ADMT 3.1 on one of those.

After getting it installed and playing around with it on a test VM, I learned a few things that helped me get all of the staff workstations migrated with minimal issues:

  • Setup a Two-Way Trust between the domains first, but be aware that if users are already authenticating on both domains by using store credentials, that may break unless you also setup permissions for users of both domains on effected shares.
  • Double-check your DNS configuration. If both domains have separate Forward Lookup Zones (which they probably do), make sure that the DNS servers in both domains are setup to perform Zone Transfers between each other, and then check to make sure that all A and PTR records are actually correct and current.
  • Make sure that the user you are logged in to on the server running ADMT is in the Domain Admins group on the target domain, and the Administrators group in primary DC on the source domain.
  • Change the DNS servers that the computers to be migrated are using to the servers on the target domain. This is important, or after the computer migrations are complete you may run in to issues when logging in (for me, Active Directory decided to continually lock out user accounts of migrated users because of a missing A record in the source domain’s DNS zone).
  • If you have any local firewall software running on the workstations that are to be migrated, either temporarily disable it or add exceptions for the Netlogon Service, File and Printer Sharing, and Windows Management Instrumentation (although the last may not strictly be needed – it was hit-or-miss for me).
  • Run the following command on the workstations that you’re migrating: net localgroup “Administrators” “DomainAdomain admins” /ADD (changing DomainA to your target domain). This is important, as local admin rights are needed for the computer migration steps.
  • If users from your source domain are using resources on your target domain and using stored credentials to authenticate, delete those stored usernames/passwords from the workstation (in most cases, open Control Panel, then User Accounts, and click ‘Manage Network Passwords’ on left). Then, once you have migrated the user accounts, give those accounts permission to access the required resources.
  • During the migration, if you are trying to migrate a computer account and you continually receive an error like ERR2:7666 Unable to access server service on the machine ‘computer.domain’.  Make sure netlogon and workstation services are running and you can authenticate yourself to the machine.  hr=0×80070005. Access is denied., and you’ve run the command above on the machine to give Domain Admins from the target domain local admin rights, you may need to remove the computer from the source domain, rejoin it to the source domain to re-establish the trust relationship, and then try the migration again.
  • After the migrations are done, make sure to go back to the DNS servers on your target domain and verify that the migrated computers’ PTR records reflect the new domain suffix (eg, changed from ‘workstation1.domainB.’ to ‘workstation1.domainA.’ (and leave the trailing . in, or you’ll have trouble!).

And that’s it! ADMT worked like a charm, and after using it to migrate and merge user accounts, and then migrate the computer accounts, everyone was off DomainB with out the hassle of needing to manually join DomainA and reconfigure the user accounts. By performing both the user account and computer account migrations, once the process was done users just had to login to their computers using ‘DomainAUsername’ instead of ‘DomainBUsername’ and everything was left exactly like it had been, right down to the desktop wallpaper.

And now I’m free to decommission two old servers.

Folder Redirection to Mapped Network Drives: Fracking Stupid

March 8, 2011 / posted in "It's a Feature", howto, Makes Sense, Microsoft, Networking, Servers, Troubleshooting, Windows / 1 Comment

While updating a set of public computers to have private file shares (making use the Home Directory account property in AD to automagically map the drive), I ran in to an issue with folder redirection. I wanted to redirect all of the standard personal folders (Documents, Pictures, Music, et al…) to the same share, so I setup folder redirection in a Group Policy Object to point those folders to the users home drive (for this example, we’ll say drive Z: was mapped to \serversharefolder).

I gave the user full rights to the share, and assigned it Owner status as well (all through the Security tab, as standard), and then configured the GPO as appropriate. After rebooting the client computer, however, I checked the Documents folder only to find that it was still pointing at the default location. A quick peek in to Event Viewer revealed the following error:

Failed to apply policy and redirect folder “Documents” to “\serversharefolder”.

Redirection options=0×80009211.

The following error occurred: “Can not create folder “\serversharefolder”".

Error details: “Access is denied.”.

Which was very strange indeed, as a brief check confirmed that yes, the domain user did in fact have full access to both the folder and the share.

Then, something I saw (and stupidly, ignored) when setting up the GPO came back to me. I fired up the GPO editor and and browsed back to the Documents folder redirection section (User ConfigurationPoliciesWindows SettingsFolder Redirection). After double-clicking the Documents option, and then switching to the Settings tab (shown below), I noticed the top two boxes (“Grant User Exclusive Rights to Documents” and “Move the Contents of Documents to the New Location”) were selected by default. Given that this was an ‘Access Denied’ error, I figured one of these two settings must be at fault, so I unchecked them.

Folder Redirection StupidityAfter rebooting the client computer, the Documents folder redirected to the Home Drive as expected.

Here’s where it gets stupid, though. On the ‘Target’ tab in the Documents properties window (visible in the screenshot above), if you have the ‘Target folder location’ set to ‘Redirect to the users home directory’, it explicitly adds a note that says “This settings ignores the value of the ‘Grant User Exclusive Rights to Documents’  option on the settings page.

Apparently not, Microsoft. Apparently not.

TL;DR Version: If Folder Redirections aren’t applying correctly, Event Viewer is showing ‘Access Denied’ messages, and you’re using Home Folders specified in the user account, disable ‘Grant User Exclusive Rights to Documents’  option on the settings page of the GPO.

Server 2008 R2 and Windows 7 Client SMB2 Share Refresh Issue

January 31, 2011 / posted in Hack, Microsoft, Networking, Servers, Software, Troubleshooting, Windows / 6 Comments

That’s one heck of a long post title, but it at least describes the issue. Here’s the setup:

  • 1x Windows Server 2008 R2 with Hyper-V/AD/File Server roles, and two shared folders. Server has dual onboard NICs, one with full access to the client network below, the other to a separate network to allow the server to be managed remotely (no gateway configured on this NIC).
  • 18x Windows 7 x86 clients
  • Standard network setup (read: no VLANs, bridging, etc…. Just one network switch).

The previous server used by these clients worked perfectly. However, upon replacing the server with the one above, my users began noticing an odd issue. If they copy one or more files/folders to a share that is visible to all of the computers, the file(s) don’t immediately show up on all of the computers – usually 3/4 of the computers will see the file(s). On the 1/4 that don’t, users either have to wait ~10 minutes before the files will appear, or they can reboot to force a refresh. Simply pressing F5, or right-clicking in the shared folder and choosing ‘Refresh’ doesn’t work – only waiting or rebooting does.

In terms of a solution, I’ve seen a number of suggestions, but none seem to work. The server has dual-onboard Broadcom Gigabit NICs, and a number of forum posts have suggested disabling Checksum Offload and Large Send Offload, but this made no difference. Neither did disabling IPv6 on the client and server side. Disabling firewalls on the client and server side made no difference, nor did this post suggesting a few registry settings to change.

What did fix the issue, though, was disabling SMB2. Once all of the clients were connecting using the old SMB protocol the issue disappeared. I have no idea why SMB2 is an issue as I haven’t take the time to troubleshoot further with SMB2-specific settings, however this at least has things running normally.

TL;DR Version: If you have clients connecting to a Windows Server 2008 R2 box and the contents of file shares aren’t refreshing immediately or until reboot, disable SMB2 on the server.

Server 2008 x64 “Please wait for the Windows Modules Installer” and Boot Looping

December 29, 2010 / posted in "It's a Feature", Microsoft, Servers, Software, Troubleshooting, Virtualization, Windows / 2 Comments

Yesterday, while running a large lump of Windows Updates, one of my Windows Server 2008 x64 boxen decided to start boot-looping in the most annoying way. After applying a group of around ten updates and restarting, the server rebooted to the Welcome Screen and hung on the message “Please wait for the Windows Modules Installer” before eventually giving up and rebooting. After that, it booted back to the Welcome Screen and this time displayed the “Configuring Updates Stage 3 of 3 0%” message. All was well, I thought, until two minutes later when the screen went black and the server (gracefully) rebooted again, and then proceeded to loop under the same conditions, regardless of which startup mode is selected (Safe Mode, Last Known Good Configuration Mode, etc… will all produce the same results).

The news, unfortunately, isn’t good. Google was little help in the case, as most of the suggestions didn’t work. The only one that did was the suggestion to rename pending.xml in C:Windowswinsxs, however this comes with a

VERY BIG,
ALL-CAPS,
RED-LETTER WARNING:

By renaming said .xml file, your server will boot. It will still briefly display the Configuring Updates message, however the login screen will appear very shortly after. The problem is that, because there are updates that have been partially installed and aborted, Windows Update is now borked (attempts to run it will only result in a 0x8000FFFF error code, which is a generic code for “Something is broken, but we don’t know what”). Based on my own experience, and that of others, there is really no way to fix it. Sorry. Microsoft KB article 946414 that suggests that this error state can be fixed by deleting the PendingXmlIdentifier value from HKEY_LOCAL_MACHINECOMPONENTS, however it doesn’t work in this case, as the entire Windows Update backend is full of half-completed operations that can’t be cleared or rolled back.

In short, if this has happened to you, I regret to inform you that your server is basically FUBAR – the only upshot of this is that as the server will now boot entirely to the desktop, and (unless something is really broken) all services should be working (note: I did have to reboot any Hyper-V virtual machines that were suspended during the reboot in order to regain network access to them). As such, you should be able to do whatever is needed to backup the server before reformatting it. Even in the event you could get Windows Update working again, I wouldn’t trust the server in a production environment.

TL;DR Version: If this happens to you, rename c:windowswinsxspending.xml using a Windows DVD to boot from, and then backup the server in prep for a reinstall.

A PHP-Based Server Monitor

October 22, 2010 / posted in Hack, howto, Servers / No Comments

The other day I decided that the little ‘Network Monitor’ desktop gadget I was using to monitor my few servers just wasn’t cutting it. Instead, I wanted to make use of a spare iMac and have something a little flashier. A Google search for Server Monitors brought up a plethora of options that were either horribly ugly, platform specific, or just didn’t work the way I needed (most required that the target server be running some form of web server, such as IIS or Apache to retrieve headers to see if the server was up – most of my servers don’t run those). As such, I decided to write a small script from scratch.

I figured the easiest way to accomplish my goal of a platform-independent monitoring script was to use PHP. After enabling Apache2/PHP5 on my Snow Leopard-running iMac (a topic for another blog post later), I searched through the PHP.net function list until I found fsockopen(). This function is quite ideal, as it will work with any open port. The first step was to make a quick function to utilize fsockopen and return some testable results:

function checkServer($ip,$port)
{
$fp = fsockopen($ip,$port,$errno,$errstr,1);
if (!$fp)
{
return ‘Down’;
} else {
return ‘Up’;
}
}

I added this to a <?php ?> block in the <head></head> of the document – to call the function and perform the test, I used the following line below:

$servername = checkServer(’192.168.1.100′,’53′);

In this example I’m checking the availability of a DNS server, so I use port 53. When this runs, the $servername is set to either ‘Up’ or ‘Down’ depending on whether or not a connection can be opened on that port.

The only thing left now was to display this output. I made a fancy table-based page with graphics where each server is a cell and the background changes between green and red depending on the $servername value. However, all you really need is the code below in a <?php ?> block in the body of the page:

echo(‘Server Example Status: ‘.$servername);

Changing Example to the name of your server. If you have more than one server to check, just make another variable, use the checkServer function to give it a value (make sure to change the IP address and use an open port!), and then add another echo line.

That’s it! To be fancy, you can add a javascript automagic page refresh to – just change the <body> tag to:

<body onLoad=”Javascript:timedRefresh(30000); display();”>

And put the following in the <head></head> section:

<script type=”text/Javascript”>

<!–

function timedRefresh(timeoutPeriod) {

setTimeout(“location.reload(true);”,timeoutPeriod);

}

//  –>

</script>

And you’re done! If the server is up, every 30 seconds your page will refresh and show:

Server Example Status: Up

IPTABLES Logging on a VPS

October 11, 2010 / posted in howto, Linux, Networking, Servers, Troubleshooting / 2 Comments

When you manage a *nix-based server, there are a few general guidelines that most admins follow; Doing things like setting a strong root password, changing SSHD to a non-standard port, and setting up logging are usually firsts. However, if you’re on a VPS, you may run in to a few issues (note that these instructions are for CentOS 5.x and may vary depending on your distro).

For example, when I was setting my the nice new VPS that I’m running this site from I attempted to enable IPTABLES logging to monitor attempts to get to the standard SSH port (22), and the port that I actually use for SSH (I won’t post the real one, but for the example I’ll use port 1234) with the following lines in “/etc/sysconfig/iptables”:

<Snip other rules>
-A INPUT -m state --state NEW -p tcp -m tcp --dport 1234 -j LOG -m limit --limit 20/m --log-level warn --log-prefix "SSH Attempt on port 1234: "
-A INPUT -p tcp -m tcp --dport 1234 -j ACCEPT
<Snip even more rules>
-A INPUT -p tcp -m tcp --dport 22 -j LOG -m limit --limit 20/m --log-level warn --log-prefix "Dropped SSH on port 22: "
-A INPUT -j DROP
Note that you need to add the LOG lines before the ACCEPT and DROP lines.  Only 20 lines will be logged per minute to prevent file sizes from going nuts in case of an attack.
After restarting IPTABLES with service iptables restart, I made a few access attempts and checked /var/log/messages — no log lines appeared, though. Then I realized I was missing something.
In “/etc/syslog.conf” I had to add the following to the end:
kern.=warn   /var/log/firewall
I opted to log to firewall instead of messages simply to keep the file clean.
I restarted SYSLOG with service syslog restart, made a few more attempts, and still nothing was appearing in “/var/log/firewall” or “/var/log/messages”. However, typing dmesg showed the relevant lines:
SSH Attempt on port 1234: IN=venet0 OUT= MAC= SRC=10.0.0.1 DST=10.0.0.2 LEN=48 TOS=0×00 PREC=0×00 TTL=116 ID=28979 DF PROTO=TCP SPT=35291 DPT=1234 WINDOW=8192 RES=0×00 SYN URGP=0
So I knew that SYSLOG was working, however it wasn’t going all the way. Then I decided to see if KLOGD was running:
[root@vps ~]# ps aux|grep klogd
root     13632  0.0  0.1   7188   788 pts/0    S+   00:07   0:00 grep klogd
So that means that KLOGD isn’t running, which is the cause of the problem! I checked “/etc/rc.d/init.d/syslog” and found that the KLOGD lines were commented out, as such:
<snip>
passed klogd skipped #daemon klogd $KLOGD_OPTIONS
<snip>
passed klogd skipped #killproc klogd
In the “start()” and “stop()” areas respectively. I simply removed the “passed klogd skipped #” parts, saved and ran service syslog restart and presto, KLOGD was up and running:
[root@vps ~]# ps aux|grep klogd
root      7542  0.0  0.0   3808   424 ?        Ss   Oct11   0:00 klogd -x
root     15402  0.0  0.1   7188   788 pts/0    S+   00:13   0:00 grep klogd
I made a few more connection attempts and verified that now everything was working correctly:
[root@vps ~]# cat /var/log/firewall
Oct 11 23:47:06 vps kernel: SSH Attempt on port 1234: IN=venet0 OUT= MAC= SRC=10.0.0.1 DST=10.0.0.2 LEN=48 TOS=0×00 PREC=0×00 TTL=116 ID=28979 DF PROTO=TCP SPT=35291 DPT=1234 WINDOW=8192 RES=0×00 SYN URGP=0
Oct 12 00:13:03 vps kernel: Dropped SSH on port 22: IN=venet0 OUT= MAC= SRC=110.77.129.166 DST=10.0.0.2 LEN=60 TOS=0×00 PREC=0×00 TTL=45 ID=59383 DF PROTO=TCP SPT=33846 DPT=22 WINDOW=5840 RES=0×00 SYN URGP=0
Done and done! IPTABLES now properly logs to “/var/log/firewall” when someone attempts to hit port 22 or 1234.
TL;DR Version: If you want IPTABLES logging enabled on your VPS, follow the normal steps to enable IPTABLES logging and then make sure KLOGD is enabled in  ”/etc/rc.d/init.d/syslog”.

Stupid Mistake: “The requested session access is denied” When Logging In

September 21, 2010 / posted in Makes Sense, Microsoft, Servers, Software, Troubleshooting, Windows / 9 Comments

I was trying to assist another admin with a login issue on a Windows Server 2008 terminal server when I encountered a slightly different login error than the one he was describing. When attempting to connect to the terminal server with a user not in the Domain Administrators security group I received the following message:

“The requested session access is denied”

The problem, it turns out, was me. When connecting, I used a desktop shortcut for Remote Desktop Connect that had the “/admin” switch applied, which instructs Remote Desktop to connect to the Console session, which is restricted to administrators only. Using a regular shortcut without said switch solved the problem.

D’oh.

Fixed: Can’t Resize Uploaded Images in WordPress

August 19, 2010 / posted in howto, Servers, Software, Troubleshooting / No Comments

Here’s one with an easy fix. If you’ve just installed WordPress on your server and can upload images but WordPress doesn’t let you resize them in the same form, SSH in to your server and do the following:

yum install php-gd

service httpd restart

And you’re done! …At least, as long as you’re using an RHEL-compatible Linux distro. If not, use your package manager of choice, or manually find and load the php-gd extension!

Logon Server Unavailable Error

August 18, 2010 / posted in Microsoft, Servers, Software, Troubleshooting, Windows / No Comments

I came back from vacation the other day to find that some computers on our primary domain (example.local) were unable to access shares on a secondary domain (test.local) located in another building, accessed via a wireless link). When attempting to open the share (or just browse to the Domain Controller), the following error would appear:

Share Error

"There are currently no logon servers available to service the logon request."

Google’ing did no good, as there were only vague references to DNS issues and WINS servers (the later of which we don’t use). As nothing had changed in the environment recently, I was at a bit of a loss. I could ping the DC (Homer) in question, and even RDP to it, but I couldn’t for the life of me access the share. NSLOOKUP behaved normally, but then I had a thought — the DC that I couldn’t access was also acting as a DNS server (the primary one for test.local) with example.local as a Secondary Zone (which, of course, contained the DNS entries for the computers that were having trouble accessing the secondary domain). When I loaded the DNS manager and clicked on that zone, I was immediately greeted with an error stating the following:

DNS Error

Turns out, there *was* a DNS problem!

The problem was that I had removed a DNS server over a year ago and it was still referenced as the primary DNS server for this zone. For some reason, the Windows DNS service had just now decided this was a problem and stopped grabbing copies of the zone from the functional secondary DNS server.

To fix this, I simply right-clicked on the zone, chose Properties, and then removed the offending server IP from the General tab and updated with the correct servers and order. As soon as I finished, the computers had no trouble accessing that DC again. Magic!

« Older Posts