A Data Center Engineers Look At Ransomware Protection

Your worst nightmare just happened.  You have just been presented with a screen popup telling you that your data is being encrypted and that you have so many days to pay.  It gets worse.  Every day you delay, chunks of your data are deleted.   It happens.  I have been on the outside – helping restore data, remove the threats, and salvage business operations and prevent further damage.  I have a unique perspective on ransomware.

We have things like Cisco AMP, a truly robust and state of the art anti-malware system that integrates at every level from the firewall to the endpoint with telemetry systems and response rates beyond anything else in the industry at a crazy fast 13 hours!  However, even Cisco will tell you that it is not possible to rely on prevention alone.  That is why they embed their AMP endpoints into everything – they keep watch for Malware that may have slipped through that 13-hour window.  If you don’t have Cisco AMP, you could have a much larger “window”, an industry average of 100 days.

So, how do you as a data center engineer protect your data when something gets through.  How do you guarantee the ability to recover and restore?  Will you be ready when that popup box shows up?

Backups

This may seem obvious, but backups are critical to a recovery.  When your data is encrypted, there is a very strong chance you will not be able to recover it using decrypting tools, so having proper backups will help.  So, what is a proper backup?  I like Veeam Backup & Replication as it has some really handy features that can make it, and its backups a little more resilient to malware.

  • Out of the box, it will backup its own configuration and repository data. If the Veeam Controller is infected, you have the ability to restore the Veeam system quickly.
  • It can replicate itself. In fact, it should if you can.  Having a secondary copy in another location can be handy – especially if you have non-real-time replication scheduled for the Controller.
  • It can backup to a variety of locations, including the cloud to protect your data from afar.
  • You can backup to deduplication appliances like Dell EMC Data Domain appliances that use protocols that are not typically prone to malware attacks.

If you don’t have Veeam, you can still achieve some of this, or all of it.  It just may not be as easy.  Either way, multi-location and multi-type backup and/or replication strategies are critical to protecting data.

SAN Snapshots

How can snapshots help?  If you are using a snapshot inclusive EMC Unity or Nimble Storage array, you can configure snapshots with retention schedules to allow you to quickly rollback to a point in time – BEFORE the malware got into the system.  You might lose a few minutes or an hour of data, but it is much better than the alternative.  If your SAN supports snapshots and you are not using them, set them up as soon as you can.

Patching

Patching is crucial.  Without patching you are more vulnerable to ransomware and malware attacks.  Typically, security patches for software are free.  A no cost or low cost measure to protect your systems, yet they are so often overlooked.  Why?  Usually it is a lack of a management system or personnel to perform the patching.  This can be addressed with things like WSUS, Microsoft System Center, or VMware Update Manager.  Use them.

Control System Access

This is more than making sure you have passwords for employees – it is often protecting employees from themselves.  You can invest in, or make use of several of the following options to restrict access to data – which is how ransomware propagates in general – through data access.

  1. Grant read-only access. If ransomware can’t write – it can’t encrypt.  Write access should be only as necessary.  This applies to databases, file systems, servers, Active Directory, etc.  You can use things like Microsoft Identity Manager to help control and automate that access.
  2. Use VDI to your advantage. Create an air-gap of sorts between your end user’s local systems and the critical systems.  Lock down folder redirection and USB redirection.  Both Citrix XenDesktop and VMware Horizon with View apply here.
  3. Use Group Policies to lock systems down. If allowing users to set a screen background worth losing all your data?
  4. Use things like Cisco ISE with posturing to ensure that only secure systems connect to the network.

You will be attacked.  That attack may not breach your systems – or it might.  Don’t say I did not warn you.  Protect your data.  Thank me later.


For the record, I work a lot with Microsoft, EMC Unity, VMware, Cisco, VCE, EMC Data Domain, Veeam, and the other technologies here.  They are what I know best, so if you feel I am biased — well, I am.  They are what I know best!

Cooking Up A Data Center — With A Salmon Recipe To Boot

Technology is an art.  If you use the wrong cables, servers, storage, switches, routers, etc. you can be sure to have a data center that puts a bad taste in your mouth.  You can achieve that bad taste by using poor quality ingredients, by assembling good or bad ingredients in the wrong way, or by designing a great system – but for the wrong purpose.  To illustrate that, let’s take a look at another art — cooking.

Cooking is the preparation of a fantastic meal through the perfect blending of raw ingredients, spices, heat, and cold.  It is also necessary to know that if you are cooking Asian food as opposed to Mexican food, you are not going to use cayenne pepper or Jalapenos generally.  Let’s take this to the extreme and cook up a data center shall we?

 

Ingredients:

  • 2 Fresh Wild Caught Salmon Fillets (Skin On One Side)
    • AKA: EMC Storage Array & Cisco UCS Servers
  • 2 Cedar Planks – Soaked in Water for 2 Hours & 2 Cedar Wraps W/Ties
    • AKA: Proper 10GB & FC Network Cabling
  • ¼ Cup Soy Sauce
    • AKA: Solid Up Line Core Network — Cisco 4500-X Switches
  • ¼ Cup Brown Sugar
    • AKA: Solid Storage Area Network – Cisco MDS Fiber Channel Switches
  • 2 Tbsp Sake
    • AKA: Solid & Stable Power
  • Salt & Black Pepper
    • AKA: A Proper, Well Tested Hypervisor Platform – VMware vSphere Baby
  • 1 Tbsp Minced Garlic
    • AKA: Proper Cable Management & Velcro
  • 1 tsp Lemon Juice
    • AKA: A Quality Security Infrastructure – Cisco ASA, FirePOWER, ISE
  • 1 BBQ Grill
    • AKA: Cisco Nexus Data Center Grade Switches

 

Now, you can go cheap, with less tried and true, potentially cheaper solutions – even stuff from the new kids on the block.  But, what are you risking?  When you use oven instead of a grill, you lose the smoked goodness it brings to the dish.  When you skip the lemon juice, it leaves your mouth desiring something more – if you do that with your security, are you leaving a gaping hole in your environment?  Skip the brown sugar and you have a tart dish that won’t move from plate to mouth very fast – kind of like what happens when you skimp on a good fiber network as opposed to iSCSI over your core network.

The point is, you have to use quality – tried and true ingredients, and mix them in the right proportions to ensure you end up with a data center dish that truly shines.  Sure, there are other brands out there besides Cisco, EMC, & VMware that make good products – okay, not sure you can beat VMware on the hypervisor aspect – but they are what I know works well most of the time; and when they don’t they have the knowledge and experience to get the taste back in balance.  Go forth – data center well, and enjoy the fruits of your labor for the next three to five years.  Do it wrong, you will be making another dish sooner than you like.

 

For those of you that want to, here is the rest of the recipe:

  1. Preheat your BBQ grill to 350 (Medium).
  2. Mix soy sauce, brown sugar, sake, garlic, and lemon juice in a bowl, set aside.
  3. Place salmon skin side down on a cedar wrap and lightly dust with salt and pepper.
  4. Place the cedar wrap on a cedar plank. Tie the wrap loosely around the salmon.
  5. Place the plank directly on the grill and BBQ for 12-15 minutes – covered.
  6. While cooking, use a spoon to generously cover the salmon with the sauce mixture. This should be done two or three times during cooking, to build up a nice glaze.
  7. The salmon will flake with a fork when ready.
  8. Eat & Enjoy.

 

Just like a good data center, this dish is sure to be mouthwatering!

EMC-Unity

First Look: EMC Unity & The Miracle Feature

A little while back, EMC announced and made available the EMC Unity Storage Array line.  Now, I am a HUGE fan of EMC and I am a bit terrified of what will be happening to the “World’s Best SANs” with the Dell takeover.  I know that Dell has not had time to really start poking around in EMC to the point where they could have made too much impact, so I was hoping that the Unity Storage Arrays would be unaffected.  It looks like I am right — either that or Dell has really surprised me.  Either way, the Unity Arrays are true works of art with all the tweaks that everyone has been looking for from the VNX/Clariion line for years.  They even threw in a few options that made me wish I had thought of them — most of them in a simplified two option software packaging program.

  • First and foremost, as the name implies, the Unity Arrays are “Unified”.  Historically, the “Unified” VNX SANs have been the bane of a storage administrator’s existence.  In the past I would rather have had all my hair pulled with a eyebrow string (what do they call those anyway) if it would mean I did not have to work on a “Unified” SAN.  Well, those days are finally over.  That is right folks — NO MORE CONTROL STATIONS OR DATA MOVERS!  When I saw this, I really did fall out of my chair.
  • One feature I am torn on is the complete lack of thick LUN support.  Everything is thin.  This just means that I will have to further emphasis that if you own a SAN — you better be monitoring it.
  • It now supports up to 64TB sized file systems with NFSv3 & 4.2 along with SMB/CIFS and SFTP/FTP multi-protocol access.  This is a big change from 16TB and it does mean that NFSv2 support is no longer.
  • The file side supports online modifiable user and tree quotas — yes, you read that correctly.
  • FAST Cache has been redesigned.  It now has a five year capacity reserve, new (I think) cache promotion methodology, and ONLINE EXPANSION AND SHRINK!

Those are some very nice and new features from the engineers at EMC, but really they are just the gravy.  Both the All-Flash and Hybrid Unity Arrays come with a feature that will delight every EMC VNX storage administrator around the world.  Perhaps the most asked for and desired feature ever requested to EMC (I don’t have statistics to prove this, but if I am wrong I will publicly apologize to EMC)…

The feature that we have all been waiting for…

The feature that will make you call your EMC Partner this very second…

The feature that will beat all other features ever introduced in any other IT product — ever…

The All New… HTML5 based — NON JAVA — GUI!

Trust me on this, I know I am right.  Its as if millions of voices suddenly cried out in joy and were instantly calling their EMC Partners.

 

Data Domain Retention Lock: Compliance Scripting

Today I had the opportunity to develop a quick and dirty PowerShell script for the EMC DataDomain Retention Lock: Compliance feature.  When using Retention lock, you have to update the last accessed date/time in order for it to trigger the retention feature on the DataDomain for any files that you want to have retained.  In my case, we are using a CIFS share and copying a bunch of files out to the share daily to be stored for a period of time — effectively using the DataDomain as a Write Once Read Many (WORM) device.  To update all the files at one time, we developed a quick script that will, once a day, scan the directory and update the access date/time.  That script is included here for reference:

#Set the directory root for the script to run.
$dirlook=”P:\”
#This is setting the script to only check files with a modified date within the last 1 day.
$backdate=$(Get-Date).AddDays(-1)
#This is the number of days to set the access date to.  Currently 7 Years.
$forwarddate=$(Get-Date).AddDays(+2555).ToString(‘MMddHHmmyyyy’)
#Find the files which are modified and modify the last access date.
Get-Childitem $dirlook -Recurse | `
where-object {!($_.psiscontainer)} | `
where { $_.LastWriteTime -gt $backdate } | `
foreach {C:\touch.exe -a -t $forwarddate $_.fullname}

Now, in order to use this, you will need to update the values a bit to mimic your requirements and you will need to get the “touch” program available here: http://sourceforge.net/projects/unxutils/?source=typ_redirect

Hope it helps you.