SAN Based Snapshots & P2V Conversion Failure

This is a post moved over from my old blog — still relevant.

I love P2V.  It works like a charm — except when it doesn’t…

When it fails, it usually fails in the strangest ways possible, the errors are obtuse, and finding the underlying cause is a nightmare.  Well, I ran into the issue where the P2V would fail referencing an error with the snapshot and/or the disk ID.  Well, I assumed this was related to the Source VSS Snapshot that is taken during the P2V process.  After spending several hours tracking that down, and realizing I was barking up the wrong tree I started looking to the destination.  Well, there was the problem.  A snapshot in the target machine.  I was pretty sure that the P2V process did not use snapshots on the target — I mean, why would it?  Time to look elsewhere.

Well, turns out that elsewhere was the SAN that was housing the storage for the target datastore.  The array, in this particular case a Nimble SAN, has a feature where you can quiescence the VMs on a LUN using the Native VMware snapshots to allow for better point in time recovery options with the SAN based protection.  If this is on, it tries to take a snapshot of all the VMs on the target LUN.  Now, if you are doing a small VM that will convert in between the snapshot window, no issue.  If it is a larger machine — turn that feature off during the P2V window and save the headache.

SharePoint Large File Library Via Windows Explorer – Error

This is a post moved over from my old blog — still relevant.

Recently, I had the opportunity to take a look at an issue with accessing SharePoint file libraries through Windows Explorer UNC shares.  When those file libraries have HUGE numbers of files, the client will hang for upwards of five minutes and then error out with the following error:

“[\\UNCLocation\] is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.  A device attached to the system is not functioning.”

Frustrating right?

Before I get to the fix, lets review the why.  When you open a folder on a Windows system (remote or local), the system has to do a couple of things:

  1. Obtain a listing of all the objects in the folder.
  2. Pull the attributes for every file.
  3. Display the files.

Well, in this case, the number of files pushes the limit of the attributes that the Windows system can load at one time due to restrictions put in place to prevent Denial of Service attacks on WebDAV Clients.  This can also happen when you are downloading VERY LARGE single files due to the same type of restrictions.

The Fix:

For Large File Libraries:

  1. Open Regedit & Go Here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\
  2. Edit the “FileAttributesLimitInBytes” value from 1000000 to 20000000

For Opening Large Files:

  1. Open Regedit & Go Here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\
  2. Edit the “FileSizeLimitInBytes” value to anything larger than the file you intend to download.  The default value is 50000000.
Cooking Up A Data Center

Cooking Up A Data Center — With A Salmon Recipe To Boot

Technology is an art.  If you use the wrong cables, servers, storage, switches, routers, etc. you can be sure to have a data center that puts a bad taste in your mouth.  You can achieve that bad taste by using poor quality ingredients, by assembling good or bad ingredients in the wrong way, or by designing a great system – but for the wrong purpose.  To illustrate that, let’s take a look at another art — cooking.

Cooking is the preparation of a fantastic meal through the perfect blending of raw ingredients, spices, heat, and cold.  It is also necessary to know that if you are cooking Asian food as opposed to Mexican food, you are not going to use cayenne pepper or Jalapenos generally.  Let’s take this to the extreme and cook up a data center shall we?

 

Ingredients:

  • 2 Fresh Wild Caught Salmon Fillets (Skin On One Side)
    • AKA: EMC Storage Array & Cisco UCS Servers
  • 2 Cedar Planks – Soaked in Water for 2 Hours & 2 Cedar Wraps W/Ties
    • AKA: Proper 10GB & FC Network Cabling
  • ¼ Cup Soy Sauce
    • AKA: Solid Up Line Core Network — Cisco 4500-X Switches
  • ¼ Cup Brown Sugar
    • AKA: Solid Storage Area Network – Cisco MDS Fiber Channel Switches
  • 2 Tbsp Sake
    • AKA: Solid & Stable Power
  • Salt & Black Pepper
    • AKA: A Proper, Well Tested Hypervisor Platform – VMware vSphere Baby
  • 1 Tbsp Minced Garlic
    • AKA: Proper Cable Management & Velcro
  • 1 tsp Lemon Juice
    • AKA: A Quality Security Infrastructure – Cisco ASA, FirePOWER, ISE
  • 1 BBQ Grill
    • AKA: Cisco Nexus Data Center Grade Switches

 

Now, you can go cheap, with less tried and true, potentially cheaper solutions – even stuff from the new kids on the block.  But, what are you risking?  When you use oven instead of a grill, you lose the smoked goodness it brings to the dish.  When you skip the lemon juice, it leaves your mouth desiring something more – if you do that with your security, are you leaving a gaping hole in your environment?  Skip the brown sugar and you have a tart dish that won’t move from plate to mouth very fast – kind of like what happens when you skimp on a good fiber network as opposed to iSCSI over your core network.

The point is, you have to use quality – tried and true ingredients, and mix them in the right proportions to ensure you end up with a data center dish that truly shines.  Sure, there are other brands out there besides Cisco, EMC, & VMware that make good products – okay, not sure you can beat VMware on the hypervisor aspect – but they are what I know works well most of the time; and when they don’t they have the knowledge and experience to get the taste back in balance.  Go forth – data center well, and enjoy the fruits of your labor for the next three to five years.  Do it wrong, you will be making another dish sooner than you like.

 

For those of you that want to, here is the rest of the recipe:

  1. Preheat your BBQ grill to 350 (Medium).
  2. Mix soy sauce, brown sugar, sake, garlic, and lemon juice in a bowl, set aside.
  3. Place salmon skin side down on a cedar wrap and lightly dust with salt and pepper.
  4. Place the cedar wrap on a cedar plank. Tie the wrap loosely around the salmon.
  5. Place the plank directly on the grill and BBQ for 12-15 minutes – covered.
  6. While cooking, use a spoon to generously cover the salmon with the sauce mixture. This should be done two or three times during cooking, to build up a nice glaze.
  7. The salmon will flake with a fork when ready.
  8. Eat & Enjoy.

 

Just like a good data center, this dish is sure to be mouthwatering!

Nuts

CCIE-DC Written In Ten Days? Am I Nuts? Probably.

Well… I am about to embark on a journey to Cisco Live!  Yep.  I set a goal several months ago that I would prepare for an exam that I would take while I was there.  Today, it comes down to the wire — I have ten days till I arrive and I have not had the opportunity to study much — if at all.  So, today I begin my challenge.  Perhaps the hardest one of my life — well other than losing some weight.  Can I?  Is it possible?

 

Can I prepare for, and pass, the written CCIE-Data Center exam in approximately ten days?  Before we take bets, let me preface this with the following caveats and statements:

  • I have extensive data center background.
  • Networking (R/S) drives me nuts.
  • I do not have my CCNA-DC or CCNP-DC.
  • I refuse to use cheats — this is against the rules.
  • I don’t really think I can do it, but damn it man — I am going to give it my damnedest try.
  • I don’t know any speed dealers — I don’t really want to either.

 

So… off to the races.  Sorry sweetie, I might be nose deep in books for a while!

Fight!  Hyper-V vs. VMware

FIGHT! The VMware vs. Hyper-V Debate Continues

Year over year, the debate continues.  Even after I write this blog post, the debate will continue.  VMware vs. Hyper-V.  The truth is that both hypervisors have their advantages and disadvantages.  To start with, let’s take a look at the prominent ones.

VMware Advantages

  • Thin hypervisor with a tiny install that can be run on a SD card.
  • FAST live migration (vMotion). This allows you to perform maintenance operations faster, without downtime.
  • Memory isolation. This is critical to prevent VM memory errors from crashing the hypervisor and vice versa.
  • Streamlined automatic dynamic memory management and transparent page sharing allowing for better consolidation ratios – to the tune +25-50% more VMs per host. It is important to note that Hyper-V does support dynamic memory management with manual configuration when all the VMs and hypervisor are on the same patch level.
  • No downtime needed to clone a VM.
  • Storage IO Control (SIOC) which is necessary to optimize storage access to VMs!
  • Dynamic serial and parallel ports.
  • Virtual Volumes & VSAN!
  • Direct driver capabilities which allow for a shorter IO path and better overall VM performance.
  • Overall better Linux, Unix, and Mac guest level support.
  • Anti-Virus offload. This is critical for VDI based deployments and helps to reduce/eliminate AV impacts to underlying disk; though we will see how this shakes out with NSX.
  • Overall Hot Add/Remove support for memory, NICs, CPUs, and disks.
  • Unified web based management through vCenter.

Hyper-V Advantages

  • Native storage support for ODX at the hypervisor level by default.
  • Network bandwidth, capping, and reservations are more flexible than Network IO Control.
  • Native clustering without central management system like vCenter.
  • Native HA without central management system like vCenter.
  • Native live migration without central management system like vCenter.

 

Really, what we have are two hypervisors that are fairly equal in basic day to day feature sets if you don’t care about consolidation ratios, high performance, and can suffer downtime to perform a large majority of management tasks – with Hyper-V.  So, if you can survive that…  Cost.

  • Hyper-V is free!  This is the one major thing that I ALWAYS hear from Hyper-V fans.  But is it really?  Hyper-V is included as part of the Windows OS – great.  Let’s not forget that VMware provides ESXi for free as well.  Granted, with the free ESXi hypervisor,  you won’t have the native cluster, HA, or Live Migration.  Also, with VMware, you do get better consolidation ratios, so you will save on the overall hardware costs since you can potentially fit more VMs on a single host.  This may not be a great thing on a single server, but if you can fit 5 Hyper-V server’s worth of VMs on a three node cluster of ESXi servers – the low cost that you pay for a base vSphere Essentials license is more than covered for in the hardware savings alone.
  • The Hyper-V management interface for a Hyper-V cluster consists of a disparate set of tools.  You need to use Failover Cluster Manager, Hyper-V Manager, and other tools just to perform basic administration tasks.  Even with SCVMM – which you will pay $10K+ for, you still can’t do full centralized management.  In a VMware environment, if I want to clone a template and spin up a VM – I am talking less than 5 minutes by clicking a wizard and assigning the customization template.  With Hyper-V I have to go through a myriad of steps that waste 20 minutes of time.  If I have to deploy 10 machines, that is no longer 50 minutes as it might be with VMware – but a total of 200 minutes with Hyper-V.  Take that across all the disconnected management tasks required and you are talking an operational cost increase of around 300% in man hours PLUS a 300% increase in maintenance windows potentially which will impact mission critical business functions.

 

I suppose if all you care about is the CAPEX cost and don’t really care about on-going OPEX costs, extended outage windows, and really feel like adding additional servers to handle your VM load while increasing power and cooling costs – well then Hyper-V is free.  VMware is not cheap, and admittedly you do have to pay for add-ons, up to a point.  Also, with VMware, the cost is upfront and renewed for support w/upgrade rights yearly (same for Hyper-V on the support if you want it).  If all you need is the basics, they both work.  If you know Hyper-V and feel like scripting PowerShell for automation, then it is quite capable.  But don’t ever tell me it is free.  Remember you mother probably told that there is nothing in this world for free – so why should you think Hyper-V is?

 

Now, I am not saying Hyper-V is bad.  But I would not use it for mission critical applications where my job depended on it.  Not yet anyway.  There may come a day.  For now, it is relegated to the lab.

EMC-Unity

First Look: EMC Unity & The Miracle Feature

A little while back, EMC announced and made available the EMC Unity Storage Array line.  Now, I am a HUGE fan of EMC and I am a bit terrified of what will be happening to the “World’s Best SANs” with the Dell takeover.  I know that Dell has not had time to really start poking around in EMC to the point where they could have made too much impact, so I was hoping that the Unity Storage Arrays would be unaffected.  It looks like I am right — either that or Dell has really surprised me.  Either way, the Unity Arrays are true works of art with all the tweaks that everyone has been looking for from the VNX/Clariion line for years.  They even threw in a few options that made me wish I had thought of them — most of them in a simplified two option software packaging program.

  • First and foremost, as the name implies, the Unity Arrays are “Unified”.  Historically, the “Unified” VNX SANs have been the bane of a storage administrator’s existence.  In the past I would rather have had all my hair pulled with a eyebrow string (what do they call those anyway) if it would mean I did not have to work on a “Unified” SAN.  Well, those days are finally over.  That is right folks — NO MORE CONTROL STATIONS OR DATA MOVERS!  When I saw this, I really did fall out of my chair.
  • One feature I am torn on is the complete lack of thick LUN support.  Everything is thin.  This just means that I will have to further emphasis that if you own a SAN — you better be monitoring it.
  • It now supports up to 64TB sized file systems with NFSv3 & 4.2 along with SMB/CIFS and SFTP/FTP multi-protocol access.  This is a big change from 16TB and it does mean that NFSv2 support is no longer.
  • The file side supports online modifiable user and tree quotas — yes, you read that correctly.
  • FAST Cache has been redesigned.  It now has a five year capacity reserve, new (I think) cache promotion methodology, and ONLINE EXPANSION AND SHRINK!

Those are some very nice and new features from the engineers at EMC, but really they are just the gravy.  Both the All-Flash and Hybrid Unity Arrays come with a feature that will delight every EMC VNX storage administrator around the world.  Perhaps the most asked for and desired feature ever requested to EMC (I don’t have statistics to prove this, but if I am wrong I will publicly apologize to EMC)…

The feature that we have all been waiting for…

The feature that will make you call your EMC Partner this very second…

The feature that will beat all other features ever introduced in any other IT product — ever…

The All New… HTML5 based — NON JAVA — GUI!

Trust me on this, I know I am right.  Its as if millions of voices suddenly cried out in joy and were instantly calling their EMC Partners.

 

Cisco HyperFlex

Cisco HyperFlex: A Zero Day Review

Cisco HyperFlex.  A converged solution from Cisco.  Scary words, right?  Wrong!

Today I had the privilege of working with Cisco on a deployment of HyperFlex.  I was expecting to run into issues and bugs galore with this being a new to market product, and I can say that I was surprised.  Everything, and I mean everything, went as smoothly as can be expected.  Not a single error.  Not one bug.  Not a single problem with the HyperFlex solution.  Out of the box, it just worked.  To keep this straight and to the point, some quick thoughts are:

  1. The HX installer is clean.  It is straightforward, easy to understand, and makes the install a nearly click…click…done scenario.  Almost.
  2. There is some planning to do upfront, and if you don’t think about the VLAN/Network structure upfront — you could be in for a bit of a setback.  While not really a big deal, proper planning for a minimum of four VLANs/Networks is needed.  Still, if you are not doing this, your are not doing it right anyway.
  3. Did I mention that their deployment tool builds all the UCS based configuration for you?  Swing and a hit!  Again, you need a little planning, but it is really a nice tool.
  4. Provisioning storage is as simple as clicking a button.
  5. I can see the design that went into this solution has taken every possible failure point into consideration, and isolated this solution from failure everywhere it can.  Of course, you have to monitor it just like any other system, but solid design is there.
  6. Cisco claims 1 hour to deploy HyperFlex.  Yes…and…No.  If you have the prerequisites in place and have the planning done ahead of time, I can see that being done in under and hour (rack/stack aside).  You could probably even do the rack/stack in that time as well, if you are a tough guy — I can’t.  🙂

I can’t say this will fit every need and every environment and SANs are not dead (oh, by the way — I am pretty sure we can add a SAN to this solution if desired — though I am not sure it is needed, but don’t quote me on that).  It is currently limited in node capacity — but unlike some naysayers out there believe, I am pretty certain this is a limitation for the initial releases.  Better to err on the side of caution than to promise more than you can deliver.  I really like that.

Good job Cisco & Springpath!

For more on HyperFlex: http://www.cisco.com/c/en/us/products/hyperconverged-infrastructure/index.html
Springpath HALO Architecture: https://vimeo.com/122110510

Interested in getting one?  Let me know, I am sure I can find someone to help you out. 😉

Load Balancing Exchange 2013 With Citrix NetScaler 11

Today, I am publishing a small guide written and intended to be used as a starting point for Load Balancing Microsoft Exchange 2013 via Citrix NetScaler 11 Build 64.34 and newer with the following expectations:

  • Provide Load Balancing (LB) to all Exchange services.
  • Provide ActiveSync Kerberos Constrained Delegation to function with iPhone, iPad (iOS Configuration Utility or AirWatch), Android (TouchDown Mail Client or AirWatch), or Windows Phone (AirWatch).
  • Provide service monitors that are in line with Microsoft best practices.
  • Provide all Exchange services via Content Switching Services (CSS) to only use one IP address.
  • Utilize responder and rewrite policies and actions to automatically redirect unsecured and root URL connections.
  • All communication from the client through to the Exchange 2013 servers will be secured.

I hope that this will be a help to the Citrix NetScaler community as a whole.  Thanks go to Rafyel G. Brooks who published a guide back in 2014 on how to deploy ActiveSync with KCD.  This guide resolves some issues with the configuration with the new NetScalers and expands on it to encompass the entire Exchange 2013 Load Balancing scenario.

Here It Is: NS11-Exchange2013-KCD-ActiveSync-Deployment

Please Enjoy!

Data Domain Retention Lock: Compliance Scripting

Today I had the opportunity to develop a quick and dirty PowerShell script for the EMC DataDomain Retention Lock: Compliance feature.  When using Retention lock, you have to update the last accessed date/time in order for it to trigger the retention feature on the DataDomain for any files that you want to have retained.  In my case, we are using a CIFS share and copying a bunch of files out to the share daily to be stored for a period of time — effectively using the DataDomain as a Write Once Read Many (WORM) device.  To update all the files at one time, we developed a quick script that will, once a day, scan the directory and update the access date/time.  That script is included here for reference:

#Set the directory root for the script to run.
$dirlook=”P:\”
#This is setting the script to only check files with a modified date within the last 1 day.
$backdate=$(Get-Date).AddDays(-1)
#This is the number of days to set the access date to.  Currently 7 Years.
$forwarddate=$(Get-Date).AddDays(+2555).ToString(‘MMddHHmmyyyy’)
#Find the files which are modified and modify the last access date.
Get-Childitem $dirlook -Recurse | `
where-object {!($_.psiscontainer)} | `
where { $_.LastWriteTime -gt $backdate } | `
foreach {C:\touch.exe -a -t $forwarddate $_.fullname}

Now, in order to use this, you will need to update the values a bit to mimic your requirements and you will need to get the “touch” program available here: http://sourceforge.net/projects/unxutils/?source=typ_redirect

Hope it helps you.