I must admit, I've become a google-fanboy over the past few years... as it's been one service I can honestly say... "it just works." It's always been reliable, and amazingly simple to use (from the user-side) but still provides full-featured APIs well documented for public use, as well as built on public standards, and a strong supporter of the open-source community... and even more amazingly... it's FREEEEEEEEEEEEEEEEEEEEEEEEE. Admittedly, they pay the bills with advertising on everything you do... and who knows what else... but honestly, the ads are subtle enough that I don't mind them... and any other avenues of revenue generation they may use I have not yet found personally intrusive. No crazy psychedelic blinky banners of doom.... which all-too-often make me never want to return to a site.
That being said, Google is also notorious for acquiring other "bits" to improve their internet-footprint. This "brain-dump" is stored on one such service. Google acquired Blogger... and after quite a while decided to integrate authentication schemes. This is exactly what caused me to almost completely lose my blog... and reevaluate my faith in Google. Truthfully... even though I was able to recover my blog... it was not due to any direct communication with Google... and was not due to any help from any of their many help-documents... and further still... was not even from their forums or the many helpful people who tech-support Google's stuff. It is much like me (an individual) trying to get Microsoft to acknowledge a bug in one of their products... without forking out crazy-amounts of cash. In-short... I followed every step they recommended to get help... and ultimately it ended up in a forum which never gets read or responded-to by anyone.
In Google's defense... I can honestly appreciate the complexities & difficulties of trying to merge two completely separate authentication systems into one. You can't simply keep both sets of usernames/passwords alive... eventually one has got to prevail. There was notice that google wanted everyone to transition to a google-account, and I thought since my account was associated with my google-apps enabled domain, everything was good-to-go. Well... sadly, I wasn't. I'm not exactly sure when Google finally flipped the switch... but there-after when I tried to log into my account I was presented with a blank blogger account. But my site still had all my posts. Browsing to my site when logged in, only showed me the traditional guest page. I tried posting on forums... password-resets... there's even an offline "recover-your-account" page which seems forever-offline with no ETA to being back-online. I was about to give up & remove DNS entries and perhaps look elsewhere for something else.
Well As you may be reading... I didn't give up. I somehow stumbled upon this page: http://www.google.com/support/accounts/bin/answer.py?answer=27443 which I didn't have a lot of faith in... but figured, what the heck? It's worth a try. Well... following the "Do you use Gmail with this account?" with the "yes" option was completely useless... but for kicks... I said no... and ended up at the "password-assistance page" (https://www.google.com/accounts/recovery) which after submitting my email address at my domain... came back with a rather strange option to send me a reset-password link... the strange part was that the email address was "myadmin%mydomain.tld@gtempaccount.com" ... well... that's not my email address... but the gtempaccount ... looked like perhaps google went & associated my blogger account with a bogus temporary account. For kicks... I logged in using my old password with the "myadmin%mydomain.tld@gtempaccount.com" as my username.... and voilà... it worked.
Just to be safe... I went and invited my proper "google-account" account with this & removed the gtempaccount.
It's a shame that there's a huge number of posts out there that also seemed to have fallen on deaf ears. I listen to many other bloggers/podcasters/etc... out there who have run into various issues with google's tools... and it seems when they have a problem google jumps as fast as they can to fix the problem ASAP... Google would be fools not-to... but the fact that I searched repeatedly across every document Google offered... followed their advise to the letter, and submitted multiple requests for help in their forums... waited over a month and still didn't get even the time of day. It seems that Google's users are only as important as the number of their followers. But hey... it's freeeeeeeeeee...
To all who have fallen into my shoes... I can truly feel your pain... it may not work for ya, but try to log-in using yourolduseracct@gtempaccount.com with your old password.
Feb 10, 2011
Hyper-V Time Sync issues... FIXED!
I've been using Microsoft Hyper-V Server for a while now, and I've run into an issue in Linux operating systems where the time would skew VERY badly. Installing NTPD didn't help, as it only updates the clock if it's within 128ms of the servers it is polling from... and running a cron job to update the time just wasn't accurate enough.
The time would skew more than a few minutes in the space of 1 hour. This is VERY unacceptable.
But! as the title suggests... there is a fix. (for me it's a simple fix... for others... may not be so simple) So, here we go!
The problem occurs because of an issue where the clock in an OS isn't based on the "hardware clock" that is kept alive by a battery... and is only sorta-accurate... but rather the OS's clock is based on a set number of cpu cycles. In a physical machine, this provides a clock that is much finer-grained than the traditional hardware-time... which is only reports whole-second increments. When you are doing highly time-sensitive things, you need a much more accurate clock. For example, VoIP (ulaw RTP audio streams) traditionally breaks audio data up into 20ms chunks of audio into 1 packet of data sends that. On the other end, those packets are put into a special sort of buffer that takes those 20ms bits and reassembles them into a continuous stream of audio. If you only had a clock accurate within 1 second... you'd have some SERIOUS delay in conversations.
Today, a VERY large number of things in computers require a highly accurate clock. Rather than each application trying to have it's own clock, operating systems provide APIs that every applications can rely on for an accurate time source. I am not 100% sure with all operating systems, but I do know that Linux has one such kernel-clock that is not based on the hardware-clock. There are kernel options that can be set to define how accurate this clock is... (1/1000ms, 1/100ms, etc....) but that's not really very relevant to this topic. In short, during the startup process, the kernel starts the os clock based on the hardware clock... and some sort of algorithm for defining a number of cpu cycles per "tick", and continues to count from there... and on shutdown sets the hardware clock to the OS's clock... Typically, in the middle, services like NTPD can make the OS's clock much more precise with regards to the actual time as defined by the NIST.
So, what goes wrong in a Virtual environment? (not just hyper-v) Well, cpu cycles are virtual. There are several different things at play all of which can make the number of cycles per tick a variable rather than a constant. Most virtual server frameworks (if not all) provide some sort of compensation to the guest OS'es to *appear* like they're getting a constant number of cycles, but this wreaks havoc if the guest OS doesn't quite understand what the host OS has done. In the case of Hyper-V, extra cpu cycles are thrown at the guest OS to try & push the clock forward periodically when it thinks the guest OS might have missed some. This *can* help, but in the case of most Linux OSes, they just steadily count the extra cpu cycles, and the OS clock skews forward. The fix? Well, this is where it gets a bit more tricky.
A Kernel Module Saves the Day!! Actually, this idea isn't as strange as it sounds. Other virtualization frameworks have "integration" tools that do exactly this... and other functions which we really aren't worried about at this point. We want Linux Guests in Hyper-V to keep time! Microsoft was sooo thoughtful to provide us with the tools we need. The "Linux Integration Services v2.1 for Windows Server 2008 Hyper-V R2" was written specifically for this purpose! We're Saved! ...or are we? Well, if you read the fine print, it's only supported in a CRAZY-short list of Linux operating systems.
SUSE Linux Enterprise Server 10 SP3 x86 and x64 (up to 4 vCPU)
SUSE Linux Enterprise Server 11 x86 and x64 (up to 4 vCPU)
Red Hat Enterprise Linux 5.2, 5.3, 5.4, and 5.5 x86 and x64 (up to 4 vCPU)
Well... that's a start, Microsoft appears to want to be friends with the Linux community... heck, they even went as far to get several of the pieces of the Linux Integration Services integrated into the kernel. Wow! Microsoft creating kernel drivers? AMAZING!.... wait... why doesn't my OS work then? Well the down side is that Microsoft managed to get the driver into the kernel, but failed to keep it there due to a GPL violation. So, you can't expect to see the hyper-v bits in any mainstream linux repositories anytime soon...
But this is not the end! This Linux Integration Services are still there and still can be useful. There's a few *gotchyas* but we are mainly focused on 1 feature... time sync.
So, without any further-ado... here's what you need to do:
1) Download the Linux Integration Services package from Microsoft, and extract the files to someplace convenient. We're only really interested in the LinuxIC v21.iso at this point.
2) Mount the .iso into your guest OS and mount the virtual cdrom to someplace convenient.
mkdir /mnt/cdrom; mount /dev/cdrom /mnt/cdrom
3) make a copy of the cdrom-stuff on the local guest OS. (the cdrom isn't writable <shock>)
mkdir /opt/linux_ic_v21_rtm; cp /mnt/cdrom/* /opt/linux_ic_v21_rtm
4) get your guest OS ready to build a kernel module. I'm using Debian 5, but your os should have something similar... (basically, just need to install the build-tools & kernel source)
apt-get install build-essential linux-source module-assistant
m-a update && m-a prepare
5) Fix one line in the script/determine_os script. Apparently, Microsoft didn't want to build the module for every kernel, just those 2.6.27 or greater. Unfortunately, I'm running 2.6.26.
This may be a bit iffy, but for my kernel, all I needed to do was change line 40 from:
if [ $KERNEL_VER -ge 27 ]
to:
if [ $KERNEL_VER -ge 26 ]
This may work for other kernel versions, and the entire script might be better modified to support other kernels, but I am not 100% sure of what is & what is not supported. I figured that 2.6.26 has very few (if any ) differences in the system clock's functions. (the entire package was designed to work with kernels 2.6.27 and kernel 2.6.9)
6) Build *only* the hv_timesource module. The other bits are very kernel-version specific. On the other side of the coin... they do contain the other nifty paravirtual drivers.... but I am not a kernel developer (or any kind of programmer) and can't tell you how to fix the compile errors.
make hv_timesource
7) If all goes well, the hv_timesource.ko module will be built! Finally, we just need to load it.
insmod src/hv_timesource.ko
Final notes: At this point, the module should be loaded, and the clock shouldn't drift anymore! That being said, it may still be wrong, so it may be useful to set it. You can use "ntpdate" or even pull the time from the hardware clock using "hwclock --hctosys". This should at least get you started, but you'll still need to make this module auto-load on startup so it will be there after reboots... and if you should upgrade the kernel version, you may need to rebuild the module manually.
I'd be a very happy person if Microsoft would split up their "integration services" package into pieces. They do have closed-source bits (which is why they were in trouble for violating the GPL) that they can keep as an add-on package... but I honestly can't see any reason why this bit should be kept from the mainstream kernel releases. This is yet another example of Microsoft playing the "see we integrate with linux" game... without actually integrating with linux.
The time would skew more than a few minutes in the space of 1 hour. This is VERY unacceptable.
But! as the title suggests... there is a fix. (for me it's a simple fix... for others... may not be so simple) So, here we go!
The problem occurs because of an issue where the clock in an OS isn't based on the "hardware clock" that is kept alive by a battery... and is only sorta-accurate... but rather the OS's clock is based on a set number of cpu cycles. In a physical machine, this provides a clock that is much finer-grained than the traditional hardware-time... which is only reports whole-second increments. When you are doing highly time-sensitive things, you need a much more accurate clock. For example, VoIP (ulaw RTP audio streams) traditionally breaks audio data up into 20ms chunks of audio into 1 packet of data sends that. On the other end, those packets are put into a special sort of buffer that takes those 20ms bits and reassembles them into a continuous stream of audio. If you only had a clock accurate within 1 second... you'd have some SERIOUS delay in conversations.
Today, a VERY large number of things in computers require a highly accurate clock. Rather than each application trying to have it's own clock, operating systems provide APIs that every applications can rely on for an accurate time source. I am not 100% sure with all operating systems, but I do know that Linux has one such kernel-clock that is not based on the hardware-clock. There are kernel options that can be set to define how accurate this clock is... (1/1000ms, 1/100ms, etc....) but that's not really very relevant to this topic. In short, during the startup process, the kernel starts the os clock based on the hardware clock... and some sort of algorithm for defining a number of cpu cycles per "tick", and continues to count from there... and on shutdown sets the hardware clock to the OS's clock... Typically, in the middle, services like NTPD can make the OS's clock much more precise with regards to the actual time as defined by the NIST.
So, what goes wrong in a Virtual environment? (not just hyper-v) Well, cpu cycles are virtual. There are several different things at play all of which can make the number of cycles per tick a variable rather than a constant. Most virtual server frameworks (if not all) provide some sort of compensation to the guest OS'es to *appear* like they're getting a constant number of cycles, but this wreaks havoc if the guest OS doesn't quite understand what the host OS has done. In the case of Hyper-V, extra cpu cycles are thrown at the guest OS to try & push the clock forward periodically when it thinks the guest OS might have missed some. This *can* help, but in the case of most Linux OSes, they just steadily count the extra cpu cycles, and the OS clock skews forward. The fix? Well, this is where it gets a bit more tricky.
A Kernel Module Saves the Day!! Actually, this idea isn't as strange as it sounds. Other virtualization frameworks have "integration" tools that do exactly this... and other functions which we really aren't worried about at this point. We want Linux Guests in Hyper-V to keep time! Microsoft was sooo thoughtful to provide us with the tools we need. The "Linux Integration Services v2.1 for Windows Server 2008 Hyper-V R2" was written specifically for this purpose! We're Saved! ...or are we? Well, if you read the fine print, it's only supported in a CRAZY-short list of Linux operating systems.
SUSE Linux Enterprise Server 10 SP3 x86 and x64 (up to 4 vCPU)
SUSE Linux Enterprise Server 11 x86 and x64 (up to 4 vCPU)
Red Hat Enterprise Linux 5.2, 5.3, 5.4, and 5.5 x86 and x64 (up to 4 vCPU)
Well... that's a start, Microsoft appears to want to be friends with the Linux community... heck, they even went as far to get several of the pieces of the Linux Integration Services integrated into the kernel. Wow! Microsoft creating kernel drivers? AMAZING!.... wait... why doesn't my OS work then? Well the down side is that Microsoft managed to get the driver into the kernel, but failed to keep it there due to a GPL violation. So, you can't expect to see the hyper-v bits in any mainstream linux repositories anytime soon...
But this is not the end! This Linux Integration Services are still there and still can be useful. There's a few *gotchyas* but we are mainly focused on 1 feature... time sync.
So, without any further-ado... here's what you need to do:
1) Download the Linux Integration Services package from Microsoft, and extract the files to someplace convenient. We're only really interested in the LinuxIC v21.iso at this point.
2) Mount the .iso into your guest OS and mount the virtual cdrom to someplace convenient.
mkdir /mnt/cdrom; mount /dev/cdrom /mnt/cdrom
3) make a copy of the cdrom-stuff on the local guest OS. (the cdrom isn't writable <shock>)
mkdir /opt/linux_ic_v21_rtm; cp /mnt/cdrom/* /opt/linux_ic_v21_rtm
4) get your guest OS ready to build a kernel module. I'm using Debian 5, but your os should have something similar... (basically, just need to install the build-tools & kernel source)
apt-get install build-essential linux-source module-assistant
m-a update && m-a prepare
5) Fix one line in the script/determine_os script. Apparently, Microsoft didn't want to build the module for every kernel, just those 2.6.27 or greater. Unfortunately, I'm running 2.6.26.
This may be a bit iffy, but for my kernel, all I needed to do was change line 40 from:
if [ $KERNEL_VER -ge 27 ]
to:
if [ $KERNEL_VER -ge 26 ]
This may work for other kernel versions, and the entire script might be better modified to support other kernels, but I am not 100% sure of what is & what is not supported. I figured that 2.6.26 has very few (if any ) differences in the system clock's functions. (the entire package was designed to work with kernels 2.6.27 and kernel 2.6.9)
6) Build *only* the hv_timesource module. The other bits are very kernel-version specific. On the other side of the coin... they do contain the other nifty paravirtual drivers.... but I am not a kernel developer (or any kind of programmer) and can't tell you how to fix the compile errors.
make hv_timesource
7) If all goes well, the hv_timesource.ko module will be built! Finally, we just need to load it.
insmod src/hv_timesource.ko
Final notes: At this point, the module should be loaded, and the clock shouldn't drift anymore! That being said, it may still be wrong, so it may be useful to set it. You can use "ntpdate" or even pull the time from the hardware clock using "hwclock --hctosys". This should at least get you started, but you'll still need to make this module auto-load on startup so it will be there after reboots... and if you should upgrade the kernel version, you may need to rebuild the module manually.
I'd be a very happy person if Microsoft would split up their "integration services" package into pieces. They do have closed-source bits (which is why they were in trouble for violating the GPL) that they can keep as an add-on package... but I honestly can't see any reason why this bit should be kept from the mainstream kernel releases. This is yet another example of Microsoft playing the "see we integrate with linux" game... without actually integrating with linux.
Subscribe to:
Posts (Atom)