A Data Center Engineers Look At Ransomware Protection

Your worst nightmare just happened.  You have just been presented with a screen popup telling you that your data is being encrypted and that you have so many days to pay.  It gets worse.  Every day you delay, chunks of your data are deleted.   It happens.  I have been on the outside – helping restore data, remove the threats, and salvage business operations and prevent further damage.  I have a unique perspective on ransomware.

We have things like Cisco AMP, a truly robust and state of the art anti-malware system that integrates at every level from the firewall to the endpoint with telemetry systems and response rates beyond anything else in the industry at a crazy fast 13 hours!  However, even Cisco will tell you that it is not possible to rely on prevention alone.  That is why they embed their AMP endpoints into everything – they keep watch for Malware that may have slipped through that 13-hour window.  If you don’t have Cisco AMP, you could have a much larger “window”, an industry average of 100 days.

So, how do you as a data center engineer protect your data when something gets through.  How do you guarantee the ability to recover and restore?  Will you be ready when that popup box shows up?

Backups

This may seem obvious, but backups are critical to a recovery.  When your data is encrypted, there is a very strong chance you will not be able to recover it using decrypting tools, so having proper backups will help.  So, what is a proper backup?  I like Veeam Backup & Replication as it has some really handy features that can make it, and its backups a little more resilient to malware.

  • Out of the box, it will backup its own configuration and repository data. If the Veeam Controller is infected, you have the ability to restore the Veeam system quickly.
  • It can replicate itself. In fact, it should if you can.  Having a secondary copy in another location can be handy – especially if you have non-real-time replication scheduled for the Controller.
  • It can backup to a variety of locations, including the cloud to protect your data from afar.
  • You can backup to deduplication appliances like Dell EMC Data Domain appliances that use protocols that are not typically prone to malware attacks.

If you don’t have Veeam, you can still achieve some of this, or all of it.  It just may not be as easy.  Either way, multi-location and multi-type backup and/or replication strategies are critical to protecting data.

SAN Snapshots

How can snapshots help?  If you are using a snapshot inclusive EMC Unity or Nimble Storage array, you can configure snapshots with retention schedules to allow you to quickly rollback to a point in time – BEFORE the malware got into the system.  You might lose a few minutes or an hour of data, but it is much better than the alternative.  If your SAN supports snapshots and you are not using them, set them up as soon as you can.

Patching

Patching is crucial.  Without patching you are more vulnerable to ransomware and malware attacks.  Typically, security patches for software are free.  A no cost or low cost measure to protect your systems, yet they are so often overlooked.  Why?  Usually it is a lack of a management system or personnel to perform the patching.  This can be addressed with things like WSUS, Microsoft System Center, or VMware Update Manager.  Use them.

Control System Access

This is more than making sure you have passwords for employees – it is often protecting employees from themselves.  You can invest in, or make use of several of the following options to restrict access to data – which is how ransomware propagates in general – through data access.

  1. Grant read-only access. If ransomware can’t write – it can’t encrypt.  Write access should be only as necessary.  This applies to databases, file systems, servers, Active Directory, etc.  You can use things like Microsoft Identity Manager to help control and automate that access.
  2. Use VDI to your advantage. Create an air-gap of sorts between your end user’s local systems and the critical systems.  Lock down folder redirection and USB redirection.  Both Citrix XenDesktop and VMware Horizon with View apply here.
  3. Use Group Policies to lock systems down. If allowing users to set a screen background worth losing all your data?
  4. Use things like Cisco ISE with posturing to ensure that only secure systems connect to the network.

You will be attacked.  That attack may not breach your systems – or it might.  Don’t say I did not warn you.  Protect your data.  Thank me later.


For the record, I work a lot with Microsoft, EMC Unity, VMware, Cisco, VCE, EMC Data Domain, Veeam, and the other technologies here.  They are what I know best, so if you feel I am biased — well, I am.  They are what I know best!

SharePoint Large File Library Via Windows Explorer – Error

This is a post moved over from my old blog — still relevant.

Recently, I had the opportunity to take a look at an issue with accessing SharePoint file libraries through Windows Explorer UNC shares.  When those file libraries have HUGE numbers of files, the client will hang for upwards of five minutes and then error out with the following error:

“[\\UNCLocation\] is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.  A device attached to the system is not functioning.”

Frustrating right?

Before I get to the fix, lets review the why.  When you open a folder on a Windows system (remote or local), the system has to do a couple of things:

  1. Obtain a listing of all the objects in the folder.
  2. Pull the attributes for every file.
  3. Display the files.

Well, in this case, the number of files pushes the limit of the attributes that the Windows system can load at one time due to restrictions put in place to prevent Denial of Service attacks on WebDAV Clients.  This can also happen when you are downloading VERY LARGE single files due to the same type of restrictions.

The Fix:

For Large File Libraries:

  1. Open Regedit & Go Here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\
  2. Edit the “FileAttributesLimitInBytes” value from 1000000 to 20000000

For Opening Large Files:

  1. Open Regedit & Go Here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\
  2. Edit the “FileSizeLimitInBytes” value to anything larger than the file you intend to download.  The default value is 50000000.

FIGHT! The VMware vs. Hyper-V Debate Continues

Year over year, the debate continues.  Even after I write this blog post, the debate will continue.  VMware vs. Hyper-V.  The truth is that both hypervisors have their advantages and disadvantages.  To start with, let’s take a look at the prominent ones.

VMware Advantages

  • Thin hypervisor with a tiny install that can be run on a SD card.
  • FAST live migration (vMotion). This allows you to perform maintenance operations faster, without downtime.
  • Memory isolation. This is critical to prevent VM memory errors from crashing the hypervisor and vice versa.
  • Streamlined automatic dynamic memory management and transparent page sharing allowing for better consolidation ratios – to the tune +25-50% more VMs per host. It is important to note that Hyper-V does support dynamic memory management with manual configuration when all the VMs and hypervisor are on the same patch level.
  • No downtime needed to clone a VM.
  • Storage IO Control (SIOC) which is necessary to optimize storage access to VMs!
  • Dynamic serial and parallel ports.
  • Virtual Volumes & VSAN!
  • Direct driver capabilities which allow for a shorter IO path and better overall VM performance.
  • Overall better Linux, Unix, and Mac guest level support.
  • Anti-Virus offload. This is critical for VDI based deployments and helps to reduce/eliminate AV impacts to underlying disk; though we will see how this shakes out with NSX.
  • Overall Hot Add/Remove support for memory, NICs, CPUs, and disks.
  • Unified web based management through vCenter.

Hyper-V Advantages

  • Native storage support for ODX at the hypervisor level by default.
  • Network bandwidth, capping, and reservations are more flexible than Network IO Control.
  • Native clustering without central management system like vCenter.
  • Native HA without central management system like vCenter.
  • Native live migration without central management system like vCenter.

 

Really, what we have are two hypervisors that are fairly equal in basic day to day feature sets if you don’t care about consolidation ratios, high performance, and can suffer downtime to perform a large majority of management tasks – with Hyper-V.  So, if you can survive that…  Cost.

  • Hyper-V is free!  This is the one major thing that I ALWAYS hear from Hyper-V fans.  But is it really?  Hyper-V is included as part of the Windows OS – great.  Let’s not forget that VMware provides ESXi for free as well.  Granted, with the free ESXi hypervisor,  you won’t have the native cluster, HA, or Live Migration.  Also, with VMware, you do get better consolidation ratios, so you will save on the overall hardware costs since you can potentially fit more VMs on a single host.  This may not be a great thing on a single server, but if you can fit 5 Hyper-V server’s worth of VMs on a three node cluster of ESXi servers – the low cost that you pay for a base vSphere Essentials license is more than covered for in the hardware savings alone.
  • The Hyper-V management interface for a Hyper-V cluster consists of a disparate set of tools.  You need to use Failover Cluster Manager, Hyper-V Manager, and other tools just to perform basic administration tasks.  Even with SCVMM – which you will pay $10K+ for, you still can’t do full centralized management.  In a VMware environment, if I want to clone a template and spin up a VM – I am talking less than 5 minutes by clicking a wizard and assigning the customization template.  With Hyper-V I have to go through a myriad of steps that waste 20 minutes of time.  If I have to deploy 10 machines, that is no longer 50 minutes as it might be with VMware – but a total of 200 minutes with Hyper-V.  Take that across all the disconnected management tasks required and you are talking an operational cost increase of around 300% in man hours PLUS a 300% increase in maintenance windows potentially which will impact mission critical business functions.

 

I suppose if all you care about is the CAPEX cost and don’t really care about on-going OPEX costs, extended outage windows, and really feel like adding additional servers to handle your VM load while increasing power and cooling costs – well then Hyper-V is free.  VMware is not cheap, and admittedly you do have to pay for add-ons, up to a point.  Also, with VMware, the cost is upfront and renewed for support w/upgrade rights yearly (same for Hyper-V on the support if you want it).  If all you need is the basics, they both work.  If you know Hyper-V and feel like scripting PowerShell for automation, then it is quite capable.  But don’t ever tell me it is free.  Remember you mother probably told that there is nothing in this world for free – so why should you think Hyper-V is?

 

Now, I am not saying Hyper-V is bad.  But I would not use it for mission critical applications where my job depended on it.  Not yet anyway.  There may come a day.  For now, it is relegated to the lab.