Aug 12, 2011

Networking issues....

Lately, I've been seeing a lot of randomly strange networking issues pop up on random things that previously worked flawlessly.  I honestly couldn't find out what was going on.  Some things remained unaffected like World of Warcraft, but games I reciently started playing like League of Legends wouldn't work even slightly.  They would appear to connect... but would just "lag" out eventually.  Some websites also suffered... including my own server hosted in Amazon's EC2.  So... what did I do?  I initially avoided problems by VPN'ing into my work network & doing whatever from there.  Doing this, I initially assumed (incorrectly) that ISP had routing issues or something along those lines.  I've been playing games that way for a few months... and it never occurred to me that my problems were NOT due to routing issues by my ISP... After all, they're a small-time local company.  Bigger companies have done far worse... and they pay technicians an exorbitant amount to "fix" such problems.  (still battling with AT&T & GTA here over another issue)

So... what was the *actual* problem? and what was the fix?

I remember in one of my old Cisco classes that an invalid MTU will result in packets that are larger than the MTU will be thrown to the bit-bucket... most common MTUs for most networks is either 1500 or 1492.  (1500 is pretty standard... but when tunneling... it's not uncommon to see 1492 because of the overhead of encapsulating the packet)

In a normal situation... you ping a server... and a response comes back.   Well... when you do a basic icmp ping in windows using the spiffy "ping" command... a 32-byte packet is sent to the remote site (not counting the overhead) and a response comes back.  For a test of MTU... you need to increase the size of the icmp packet to the actual size of the MTU ... and tell it not to fragment.

C:\> ping -f -l 1500

well guess what?

Packet needs to be fragmented but DF set.

Hmmm.... well lets try a smaller packet

C:\> ping -f -l 1492
Packet needs to be fragmented but DF set.

ok... that's interesting... lets try smaller-still...

C:\> ping -f -l 1400
Reply from bytes=1400 time=58ms TTL=54

ok... so 1400 worked... lets keep trying bigger values until we discover what actually works...  In my case... everything up to and including 1464 worked... but nothing bigger.  Well... what's the MTU set on the interface in windows 7?.... 

C:\> netsh interface ipv4 show subinterfaces
   MTU  MediaSenseState   Bytes In  Bytes Out  Interface
------  ---------------  ---------  ---------  -------------
4294967295                1          0     393572  Loopback Pseudo-Interface 1
  1504                1  196616510   13092640  Local Area Connection 3

1504??? wow.... that's not going to work frequently... lets fix that... (requires an administrative cmd prompt)

C:\> netsh interface ipv4 set subinterface "Local Area Connection 3" mtu=1464 store=persistent

Lets try connecting to stuff & see if that helps.    Hey!!!  Everything works now!  So... what have we learned? ... there's an unspoken *standard* MTU that almost everything uses... it's 1500.  When tunneling traffic... it's 1492 (like for VPNs or PPPoE like used in DSL).   This does NOT mean that your ISP *will* permit packets of that size to reach the Internet.  It might end up being configured to work in a double-tunnel... (starts at 1500... some sort of tunnel to DSLAM... and another tunnel to endpoint... ) or there might be a combination of several things.  Windows supposedly has some sort of mechanic for adjusting the MTU on the fly... but that can fail... (like in my case).   There is nothing that will directly indicate that your MTU is set too high... other than randomly you'll see connections go "stale" on the fly... without an indication that something is wrong.  *some* routers will allow you to manually set the MTU... but I was not so lucky.  So... the final result... I set the MTU manually... and now everything works.

You might run into this same situation... it's very difficult to diagnose... as everyone just assumes a ping is a good indication that your connection is working perfectly.  Now that more & more network appliances and adapters are supporting jumbo frames (mtu ~9000) ... there's going to be more situations where discovering the MTU will fail... and this situation will need to be addressed.  Perhaps this will end up being a useful resource to others.

Feb 11, 2011

WOOHOOO! I recovered my blogger account! (no thanks to google)

I must admit, I've become a google-fanboy over the past few years... as it's been one service I can honestly say... "it just works." It's always been reliable, and amazingly simple to use (from the user-side) but still provides full-featured APIs well documented for public use, as well as built on public standards, and a strong supporter of the open-source community... and even more amazingly... it's FREEEEEEEEEEEEEEEEEEEEEEEEE.  Admittedly, they pay the bills with advertising on everything you do... and who knows what else... but honestly, the ads are subtle enough that I don't mind them... and any other avenues of revenue generation they may use I have not yet found personally intrusive.  No crazy psychedelic blinky banners of doom.... which all-too-often make me never want to return to a site.
That being said, Google is also notorious for acquiring other "bits" to improve their internet-footprint.  This "brain-dump" is stored on one such service.  Google acquired Blogger... and after quite a while decided to integrate authentication schemes.  This is exactly what caused me to almost completely lose my blog... and reevaluate my faith in Google.  Truthfully... even though I was able to recover my blog... it was not due to any direct communication with Google... and was not due to any help from any of their many help-documents... and further still... was not even from their forums or the many helpful people who tech-support Google's stuff.  It is much like me (an individual) trying to get Microsoft to acknowledge a bug in one of their products... without forking out crazy-amounts of cash. In-short... I followed every step they recommended to get help... and ultimately it ended up in a forum which never gets read or responded-to by anyone.  
In Google's defense... I can honestly appreciate the complexities & difficulties of trying to merge two completely separate authentication systems into one.  You can't simply keep both sets of usernames/passwords alive... eventually one has got to prevail.  There was notice that google wanted everyone to transition to a google-account, and I thought since my account was associated with my google-apps enabled domain, everything was good-to-go.  Well... sadly, I wasn't.  I'm not exactly sure when Google finally flipped the switch... but there-after when I tried to log into my account I was presented with a blank blogger account.  But my site still had all my posts.  Browsing to my site when logged in, only showed me the traditional guest page.  I tried posting on forums... password-resets... there's even an offline "recover-your-account" page which seems forever-offline with no ETA to being back-online.  I was about to give up & remove DNS entries and perhaps look elsewhere for something else.
Well As you may be reading... I didn't give up.  I somehow stumbled upon this page: which I didn't have a lot of faith in... but figured, what the heck?  It's worth a try.  Well... following the "Do you use Gmail with this account?" with the "yes" option was completely useless... but for kicks... I said no... and ended up at the "password-assistance page" ( which after submitting my email address at my domain... came back with a rather strange option to send me a reset-password link... the strange part was that the email address was ""   ... well... that's not my email address... but the gtempaccount ... looked like perhaps google went & associated my blogger account with a bogus temporary account.  For kicks... I logged in using my old password with the "" as my username.... and voilĂ ... it worked.
Just to be safe... I went and invited my proper "google-account" account with this & removed the gtempaccount.
It's a shame that there's a huge number of posts out there that also seemed to have fallen on deaf ears.  I listen to many other bloggers/podcasters/etc... out there who have run into various issues with google's tools... and it seems when they have a problem google jumps as fast as they can to fix the problem ASAP... Google would be fools not-to... but the fact that I searched repeatedly across every document Google offered...  followed their advise to the letter, and submitted multiple requests for help in their forums... waited over a month and still didn't get even the time of day.  It seems that Google's users are only as important as the number of their followers.  But hey... it's freeeeeeeeeee...

To all who have fallen into my shoes... I can truly feel your pain... it may not work for ya, but try to log-in using with your old password.

Feb 10, 2011

Hyper-V Time Sync issues... FIXED!

I've been using Microsoft Hyper-V Server for a while now, and I've run into an issue in Linux operating systems where the time would skew VERY badly.  Installing NTPD didn't help, as it only updates the clock if it's within 128ms of the servers it is polling from... and running a cron job to update the time just wasn't accurate enough.
The time would skew more than a few minutes in the space of 1 hour.  This is VERY unacceptable.

But!  as the title suggests... there is a fix.  (for me it's a simple fix... for others... may not be so simple)  So, here we go!

The problem occurs because of an issue where the clock in an OS isn't based on the "hardware clock" that is kept alive by a battery... and is only sorta-accurate... but rather the OS's clock is based on a set number of cpu cycles.  In a physical machine, this provides a clock that is much finer-grained than the traditional hardware-time... which is only reports whole-second increments.  When you are doing highly time-sensitive things, you need a much more accurate clock.  For example,  VoIP (ulaw RTP audio streams) traditionally breaks audio data up into 20ms chunks of audio into 1 packet of data sends that.  On the other end, those packets are put into a special sort of buffer that takes those 20ms bits and reassembles them into a continuous stream of audio.  If you only had a clock accurate within 1 second... you'd have some SERIOUS delay in conversations.

Today, a VERY large number of things in computers require a highly accurate clock.  Rather than each application trying to have it's own clock, operating systems provide APIs that every applications can rely on for an accurate time source.  I am not 100% sure with all operating systems, but I do know that Linux has one such kernel-clock that is not based on the hardware-clock.  There are kernel options that can be set to define how accurate this clock is... (1/1000ms, 1/100ms, etc....) but that's not really very relevant to this topic.  In short, during the startup process, the kernel starts the os clock based on the hardware clock... and some sort of algorithm for defining a number of cpu cycles per "tick", and continues to count from there... and on shutdown sets the hardware clock to the OS's clock...  Typically, in the middle, services like NTPD can make the OS's clock much more precise with regards to the actual time as defined by the NIST.

So, what goes wrong in a Virtual environment? (not just hyper-v)  Well, cpu cycles are virtual.  There are several different things at play all of which can make the number of cycles per tick a variable rather than a constant.   Most virtual server frameworks (if not all) provide some sort of compensation to the guest OS'es to *appear* like they're getting a constant number of cycles, but this wreaks havoc if the guest OS doesn't quite understand what the host OS has done.  In the case of Hyper-V, extra cpu cycles are thrown at the guest OS to try & push the clock forward periodically when it thinks the guest OS might have missed some.  This *can* help, but in the case of most Linux OSes, they just steadily count the extra cpu cycles, and the OS clock skews forward.  The fix? Well, this is where it gets a bit more tricky.

A Kernel Module Saves the Day!!  Actually, this idea isn't as strange as it sounds.  Other virtualization frameworks have "integration" tools that do exactly this... and other functions which we really aren't worried about at this point.  We want Linux Guests in Hyper-V to keep time!  Microsoft was sooo thoughtful to provide us with the tools we need.  The "Linux Integration Services v2.1 for Windows Server 2008 Hyper-V R2" was written specifically for this purpose!  We're Saved!  ...or are we?  Well, if you read the fine print, it's only supported in a CRAZY-short list of Linux operating systems.

SUSE Linux Enterprise Server 10 SP3 x86 and x64 (up to 4 vCPU)
SUSE Linux Enterprise Server 11 x86 and x64 (up to 4 vCPU)
Red Hat Enterprise Linux 5.2, 5.3, 5.4, and 5.5 x86 and x64 (up to 4 vCPU)
Well... that's a start, Microsoft appears to want to be friends with the Linux community... heck, they even went as far to get several of the pieces of the Linux Integration Services integrated into the kernel.  Wow!  Microsoft creating kernel drivers?   AMAZING!.... wait... why doesn't my OS work then?  Well the down side is that Microsoft managed to get the driver into the kernel, but failed to keep it there due to a GPL violation.  So, you can't expect to see the hyper-v bits in any mainstream linux repositories anytime soon...

But this is not the end!  This Linux Integration Services are still there and still can be useful.  There's a few *gotchyas* but we are mainly focused on 1 feature... time sync.

So, without any further-ado... here's what you need to do:

1) Download the Linux Integration Services package from Microsoft, and extract the files to someplace convenient.  We're only really interested in the LinuxIC v21.iso at this point.

2) Mount the .iso into your guest OS and mount the virtual cdrom to someplace convenient.
mkdir /mnt/cdrom; mount /dev/cdrom /mnt/cdrom
3) make a copy of the cdrom-stuff on the local guest OS.  (the cdrom isn't writable <shock>)
mkdir /opt/linux_ic_v21_rtm; cp /mnt/cdrom/* /opt/linux_ic_v21_rtm
4) get your guest OS ready to build a kernel module.  I'm using Debian 5, but your os should have something similar... (basically, just need to install the build-tools & kernel source)
apt-get install build-essential linux-source module-assistant
m-a update && m-a prepare
5) Fix one line in the script/determine_os script.  Apparently, Microsoft didn't want to build the module for every kernel, just those 2.6.27 or greater.  Unfortunately, I'm running 2.6.26.

This may be a bit iffy, but for my kernel, all I needed to do was change line 40 from:
if [ $KERNEL_VER -ge 27 ]
if [ $KERNEL_VER -ge 26 ]
This may work for other kernel versions, and the entire script might be better modified to support other kernels, but I am not 100% sure of what is & what is not supported.  I figured that 2.6.26 has very few (if any  ) differences in the system clock's functions.  (the entire package was designed to work with kernels 2.6.27 and kernel 2.6.9)

6)  Build *only* the hv_timesource module.  The other bits are very kernel-version specific.  On the other side of the coin... they do contain the other nifty paravirtual drivers.... but I am not a kernel developer (or any kind of programmer) and can't tell you how to fix the compile errors.
make hv_timesource
7)  If all goes well, the hv_timesource.ko module will be built!  Finally, we just need to load it.
insmod src/hv_timesource.ko
Final notes:  At this point, the module should be loaded, and the clock shouldn't drift anymore!  That being said, it may still be wrong, so it may be useful to set it.  You can use "ntpdate" or even pull the time from the hardware clock using "hwclock --hctosys".  This should at least get you started, but you'll still need to make this module auto-load on startup so it will be there after reboots... and if you should upgrade the kernel version, you may need to rebuild the module manually.

I'd be a very happy person if Microsoft would split up their "integration services" package into pieces.  They do have closed-source bits (which is why they were in trouble for violating the GPL) that they can keep as an add-on package... but I honestly can't see any reason why this bit should be kept from the mainstream kernel releases.  This is yet another example of Microsoft playing the "see we integrate with linux" game... without actually integrating with linux.