A Data Center Engineers Look At Ransomware Protection

Your worst nightmare just happened.  You have just been presented with a screen popup telling you that your data is being encrypted and that you have so many days to pay.  It gets worse.  Every day you delay, chunks of your data are deleted.   It happens.  I have been on the outside – helping restore data, remove the threats, and salvage business operations and prevent further damage.  I have a unique perspective on ransomware.

We have things like Cisco AMP, a truly robust and state of the art anti-malware system that integrates at every level from the firewall to the endpoint with telemetry systems and response rates beyond anything else in the industry at a crazy fast 13 hours!  However, even Cisco will tell you that it is not possible to rely on prevention alone.  That is why they embed their AMP endpoints into everything – they keep watch for Malware that may have slipped through that 13-hour window.  If you don’t have Cisco AMP, you could have a much larger “window”, an industry average of 100 days.

So, how do you as a data center engineer protect your data when something gets through.  How do you guarantee the ability to recover and restore?  Will you be ready when that popup box shows up?

Backups

This may seem obvious, but backups are critical to a recovery.  When your data is encrypted, there is a very strong chance you will not be able to recover it using decrypting tools, so having proper backups will help.  So, what is a proper backup?  I like Veeam Backup & Replication as it has some really handy features that can make it, and its backups a little more resilient to malware.

  • Out of the box, it will backup its own configuration and repository data. If the Veeam Controller is infected, you have the ability to restore the Veeam system quickly.
  • It can replicate itself. In fact, it should if you can.  Having a secondary copy in another location can be handy – especially if you have non-real-time replication scheduled for the Controller.
  • It can backup to a variety of locations, including the cloud to protect your data from afar.
  • You can backup to deduplication appliances like Dell EMC Data Domain appliances that use protocols that are not typically prone to malware attacks.

If you don’t have Veeam, you can still achieve some of this, or all of it.  It just may not be as easy.  Either way, multi-location and multi-type backup and/or replication strategies are critical to protecting data.

SAN Snapshots

How can snapshots help?  If you are using a snapshot inclusive EMC Unity or Nimble Storage array, you can configure snapshots with retention schedules to allow you to quickly rollback to a point in time – BEFORE the malware got into the system.  You might lose a few minutes or an hour of data, but it is much better than the alternative.  If your SAN supports snapshots and you are not using them, set them up as soon as you can.

Patching

Patching is crucial.  Without patching you are more vulnerable to ransomware and malware attacks.  Typically, security patches for software are free.  A no cost or low cost measure to protect your systems, yet they are so often overlooked.  Why?  Usually it is a lack of a management system or personnel to perform the patching.  This can be addressed with things like WSUS, Microsoft System Center, or VMware Update Manager.  Use them.

Control System Access

This is more than making sure you have passwords for employees – it is often protecting employees from themselves.  You can invest in, or make use of several of the following options to restrict access to data – which is how ransomware propagates in general – through data access.

  1. Grant read-only access. If ransomware can’t write – it can’t encrypt.  Write access should be only as necessary.  This applies to databases, file systems, servers, Active Directory, etc.  You can use things like Microsoft Identity Manager to help control and automate that access.
  2. Use VDI to your advantage. Create an air-gap of sorts between your end user’s local systems and the critical systems.  Lock down folder redirection and USB redirection.  Both Citrix XenDesktop and VMware Horizon with View apply here.
  3. Use Group Policies to lock systems down. If allowing users to set a screen background worth losing all your data?
  4. Use things like Cisco ISE with posturing to ensure that only secure systems connect to the network.

You will be attacked.  That attack may not breach your systems – or it might.  Don’t say I did not warn you.  Protect your data.  Thank me later.


For the record, I work a lot with Microsoft, EMC Unity, VMware, Cisco, VCE, EMC Data Domain, Veeam, and the other technologies here.  They are what I know best, so if you feel I am biased — well, I am.  They are what I know best!

Musings On Employment

I love where I work.  I have liked my job before.  I have liked the people I work with before.  I have hated where I worked before.  I have been way underpaid before.  I have been laid-off before.  I have been on all sides of the spectrum.  I have never, been able to say that I love where I work — up until now.  Many times I have mused over why.  I can’t be certain that I speak for everyone — as not everyone can thrive in the same work environments, but I can say that there is a distinct set of differences that add up to make my current employer the best place I have ever worked.  If you want to learn a few lessons from them, here are some take-away thoughts:

  • Challenge: The work is very challenging at times.  We get to work on all kinds of technologies, at every skill level.  We are constantly dabbling in a variety of technologies, with a variety of customer types and people.  I have the type of mind that enjoys being tried and challenged — this works for me.  Employees thrive with challenge in my experience.
  • Opportunity:  There is always opportunity to participate is company projects, helping to lead change, and to even dig in on projects and problems that can arise.  I can choose how much — or how little — I can volunteer for.  Make those options available, and keep your doors open — literally and figuratively.
  • Work/Life Balance:  I know.  You are thinking that most companies claim to offer this.  Well, I have been at some of those companies and I can tell you, I understand — most don’t.  What I do know is that currently, and for the last several years, I work.  We all do.  However, the company consciously makes efforts to ensure that we are not overworked, that we can take time when we must, and that we are able to put family on the pedestal they deserve.  There is a balance, and while we may not get every request we ask for, the management I deal with tries exceptionally hard to make sure we do.
  • Fairness:  The management team where I work, and who I report up to, have a habit of being fair — and trying hard to be.  Be fair to the company and the company will be fair to you.  Make that your motto.
  • People:  This is the most important one to me.  Everyone in the office has mutual respect for one another.  I think this comes from having some of the best and brightest minds on staff.  When you earn the respect, instead of demand it, it just makes for better relationships.  Learn to make sure the people that work for you are smarter than you are, that you foster teamwork and collaboration, that you allow a little play, and you will be in a great place.
  • Training:  This goes without saying.  Don’t promise it and never deliver.  Take a page out of my employer’s book and deliver on it.  It does not have to be formal training, but there should be some.  Balance on the job training, shadowing, collaboration, video training, classroom, and self-study.  It works.
  • Benefits:  Don’t pinch pennies here.  Ever.

Ultimately, I find that if you keep evaluating yourself (and the company) and look to be better, you will be.  My current employer has earned my service and I hope to serve them well into the future.  Help your employees feel the same way.  If you don’t they might just end up working for my employer.