A Data Center Engineers Look At Ransomware Protection

Your worst nightmare just happened.  You have just been presented with a screen popup telling you that your data is being encrypted and that you have so many days to pay.  It gets worse.  Every day you delay, chunks of your data are deleted.   It happens.  I have been on the outside – helping restore data, remove the threats, and salvage business operations and prevent further damage.  I have a unique perspective on ransomware.

We have things like Cisco AMP, a truly robust and state of the art anti-malware system that integrates at every level from the firewall to the endpoint with telemetry systems and response rates beyond anything else in the industry at a crazy fast 13 hours!  However, even Cisco will tell you that it is not possible to rely on prevention alone.  That is why they embed their AMP endpoints into everything – they keep watch for Malware that may have slipped through that 13-hour window.  If you don’t have Cisco AMP, you could have a much larger “window”, an industry average of 100 days.

So, how do you as a data center engineer protect your data when something gets through.  How do you guarantee the ability to recover and restore?  Will you be ready when that popup box shows up?

Backups

This may seem obvious, but backups are critical to a recovery.  When your data is encrypted, there is a very strong chance you will not be able to recover it using decrypting tools, so having proper backups will help.  So, what is a proper backup?  I like Veeam Backup & Replication as it has some really handy features that can make it, and its backups a little more resilient to malware.

  • Out of the box, it will backup its own configuration and repository data. If the Veeam Controller is infected, you have the ability to restore the Veeam system quickly.
  • It can replicate itself. In fact, it should if you can.  Having a secondary copy in another location can be handy – especially if you have non-real-time replication scheduled for the Controller.
  • It can backup to a variety of locations, including the cloud to protect your data from afar.
  • You can backup to deduplication appliances like Dell EMC Data Domain appliances that use protocols that are not typically prone to malware attacks.

If you don’t have Veeam, you can still achieve some of this, or all of it.  It just may not be as easy.  Either way, multi-location and multi-type backup and/or replication strategies are critical to protecting data.

SAN Snapshots

How can snapshots help?  If you are using a snapshot inclusive EMC Unity or Nimble Storage array, you can configure snapshots with retention schedules to allow you to quickly rollback to a point in time – BEFORE the malware got into the system.  You might lose a few minutes or an hour of data, but it is much better than the alternative.  If your SAN supports snapshots and you are not using them, set them up as soon as you can.

Patching

Patching is crucial.  Without patching you are more vulnerable to ransomware and malware attacks.  Typically, security patches for software are free.  A no cost or low cost measure to protect your systems, yet they are so often overlooked.  Why?  Usually it is a lack of a management system or personnel to perform the patching.  This can be addressed with things like WSUS, Microsoft System Center, or VMware Update Manager.  Use them.

Control System Access

This is more than making sure you have passwords for employees – it is often protecting employees from themselves.  You can invest in, or make use of several of the following options to restrict access to data – which is how ransomware propagates in general – through data access.

  1. Grant read-only access. If ransomware can’t write – it can’t encrypt.  Write access should be only as necessary.  This applies to databases, file systems, servers, Active Directory, etc.  You can use things like Microsoft Identity Manager to help control and automate that access.
  2. Use VDI to your advantage. Create an air-gap of sorts between your end user’s local systems and the critical systems.  Lock down folder redirection and USB redirection.  Both Citrix XenDesktop and VMware Horizon with View apply here.
  3. Use Group Policies to lock systems down. If allowing users to set a screen background worth losing all your data?
  4. Use things like Cisco ISE with posturing to ensure that only secure systems connect to the network.

You will be attacked.  That attack may not breach your systems – or it might.  Don’t say I did not warn you.  Protect your data.  Thank me later.


For the record, I work a lot with Microsoft, EMC Unity, VMware, Cisco, VCE, EMC Data Domain, Veeam, and the other technologies here.  They are what I know best, so if you feel I am biased — well, I am.  They are what I know best!

HyperFlex: An Enhanced Look

This is a post originally written for the company blog — posted here for posterity.

In the IT industry, the phrase “we are pretty much a 100% physical shop” is one that you dread to hear – especially from a fast-growing company. Such was the case with a leader in the financial services industry recently when they asked Sentinel to install a Virtual Desktop Infrastructure (VDI) solution for a new call center rollout of around 250 desktops as well as fully re-deploy their physical desktop and server infrastructures. They were pretty set on a hyper-converged solution and were looking for something scalable and easy to manage. To be successful, in the eyes of the business, the solution had to:

  1. Be solid. With internal hesitation to virtualization from the business, there had to be reliability.
  2. Be fast to deploy. To meet the aggressive deadlines, there could be zero delay on delivery or deployment.
  3. Be lightning fast. To aid in business buy-in and adoption, the solution had to deliver a better end-user experience than the current desktops. Performance was critical to that.

After reviewing the vendor options, the customer ultimately chose Cisco HyperFlex and VMware Horizon for their hyper-converged VDI solution. Aggressive deployment timelines were set and equipment was on the way. From there we moved onto the fun stuff.

The HyperFlex cluster was delivered quickly. Really quickly. Once the gear was on-site it was time to deploy. Before we go there, I want to touch on one particular aspect of the solution. Sentinel knows that maintaining data integrity and availability is essential to our customers as they adopt and adapt to new technology. How the Cisco HyperFlex solution delivers that can be summed up pretty easily:

  • The Cisco HyperFlex product line is a variant of the Unified Computing System (UCS) product line, and with that you have the full redundant design of dual fabric interconnects, full multi-pathing, and server hardware that is designed with zero single point of failure. In this particular deployment, we had four nodes (N+1) with dual fabric interconnects, and two 10GB paths from each of the HX240c nodes. Everything also ran on fully redundant power. It was a strong platform to begin from.
  • The SpringPath HALO Architecture is a file system – I am simplifying things here a bit – that allows for distribution of writes onto multiple solid-state drives (SSDs) across multiple nodes BEFORE acknowledging the writes. This maintains the data integrity by ensuring that there are multiple copies of the data on separate nodes in the cluster to prevent potential data loss.
  • The HALO Architecture enhances the data integrity by using a Log Structured Distributed Object Store to allocate the data as small objects across multiple servers in a sequential pattern, which are in turn replicated to other pool members to achieve data redundancy. By doing so, they increase not only performance, but the life of the flash layer disk in the servers as well as redundancy overall.

Back to the deployment. In a post on my personal blog, I mentioned that the HyperFlex deployment was pretty fast. Once you rack and cable the cluster, the HX installer is a breeze. What I love about the HX installer is the fact that it really does build the entire UCS deployment and makes adding a node to an existing cluster just as easy. Click. Click. Done. Overall, the deployment of the HX system after rack and cable took less time than installing the vCenter server that was required for the deployment (Note: The vCenter must be on separate hardware but can be moved into the HyperFlex cluster for ongoing operations).

After meeting the first two objectives, we needed to look at the speed. Since this was a VDI cluster, we made one small change (one line in a configuration file) to optimize the cluster’s L3 Cache for a read-heavy environment. Once that small change was made, it was time to run some tests. Since Sentinel doesn’t own the environment I will only include the following observations:

  • During testing of the 4-Node cluster with 4xVMs pushing I/O, the cluster achieved well over 125,000 I/Ops. Even in the worst-case boot storm of 250 users logging in within a one-minute period you would only really require 117,500 I/Ops, leaving plenty of room to spare. Keep in mind, this was not done in a controlled lab under ideal circumstances.
  • I was able to clone a 100GB (65 Used Thin) VM from template in less than three seconds. Seriously.
  • I deployed 250 linked clone desktops including two boots, customization, and domain join in under seven minutes. The bottleneck was the VDI limit on the maximum concurrent operations sent to vCenter (which I tweaked to 25) and probably the Active Directory domain join tasks as part of the customization. It was fun watching the vCenter task pane roll by so fast I couldn’t keep up with it.

The customer was extremely happy with the performance, scalability and easy management of their new infrastructure. The Cisco HyperFlex and VMware Horizon solution met the requirements so well that I better understand the hype around Cisco HyperFlex and the SpringPath HALO Architecture.

Of further interest in terms of scalability comes confirmation from Cisco that node capacity expansion beyond the current self-imposed limitation is in the works and will not be limited to hardware. External storage is also fully supported. This means you will have the capability to hyper-converge your core systems and still make use of external storage area networks (SAN) where business needs dictate.

All in all, HyperFlex is a rock solid platform with a fantastic and robust architecture that you would be wise to evaluate. Couple it with VMware Horizon for desktop deployment, and you have an infrastructure built to help your business achieve unprecedented levels of success. If you would like to learn more about HyperFlex or other converged/hyper-converged infrastructure solutions, please contact Sentinel for more information.

Cooking Up A Data Center — With A Salmon Recipe To Boot

Technology is an art.  If you use the wrong cables, servers, storage, switches, routers, etc. you can be sure to have a data center that puts a bad taste in your mouth.  You can achieve that bad taste by using poor quality ingredients, by assembling good or bad ingredients in the wrong way, or by designing a great system – but for the wrong purpose.  To illustrate that, let’s take a look at another art — cooking.

Cooking is the preparation of a fantastic meal through the perfect blending of raw ingredients, spices, heat, and cold.  It is also necessary to know that if you are cooking Asian food as opposed to Mexican food, you are not going to use cayenne pepper or Jalapenos generally.  Let’s take this to the extreme and cook up a data center shall we?

 

Ingredients:

  • 2 Fresh Wild Caught Salmon Fillets (Skin On One Side)
    • AKA: EMC Storage Array & Cisco UCS Servers
  • 2 Cedar Planks – Soaked in Water for 2 Hours & 2 Cedar Wraps W/Ties
    • AKA: Proper 10GB & FC Network Cabling
  • ¼ Cup Soy Sauce
    • AKA: Solid Up Line Core Network — Cisco 4500-X Switches
  • ¼ Cup Brown Sugar
    • AKA: Solid Storage Area Network – Cisco MDS Fiber Channel Switches
  • 2 Tbsp Sake
    • AKA: Solid & Stable Power
  • Salt & Black Pepper
    • AKA: A Proper, Well Tested Hypervisor Platform – VMware vSphere Baby
  • 1 Tbsp Minced Garlic
    • AKA: Proper Cable Management & Velcro
  • 1 tsp Lemon Juice
    • AKA: A Quality Security Infrastructure – Cisco ASA, FirePOWER, ISE
  • 1 BBQ Grill
    • AKA: Cisco Nexus Data Center Grade Switches

 

Now, you can go cheap, with less tried and true, potentially cheaper solutions – even stuff from the new kids on the block.  But, what are you risking?  When you use oven instead of a grill, you lose the smoked goodness it brings to the dish.  When you skip the lemon juice, it leaves your mouth desiring something more – if you do that with your security, are you leaving a gaping hole in your environment?  Skip the brown sugar and you have a tart dish that won’t move from plate to mouth very fast – kind of like what happens when you skimp on a good fiber network as opposed to iSCSI over your core network.

The point is, you have to use quality – tried and true ingredients, and mix them in the right proportions to ensure you end up with a data center dish that truly shines.  Sure, there are other brands out there besides Cisco, EMC, & VMware that make good products – okay, not sure you can beat VMware on the hypervisor aspect – but they are what I know works well most of the time; and when they don’t they have the knowledge and experience to get the taste back in balance.  Go forth – data center well, and enjoy the fruits of your labor for the next three to five years.  Do it wrong, you will be making another dish sooner than you like.

 

For those of you that want to, here is the rest of the recipe:

  1. Preheat your BBQ grill to 350 (Medium).
  2. Mix soy sauce, brown sugar, sake, garlic, and lemon juice in a bowl, set aside.
  3. Place salmon skin side down on a cedar wrap and lightly dust with salt and pepper.
  4. Place the cedar wrap on a cedar plank. Tie the wrap loosely around the salmon.
  5. Place the plank directly on the grill and BBQ for 12-15 minutes – covered.
  6. While cooking, use a spoon to generously cover the salmon with the sauce mixture. This should be done two or three times during cooking, to build up a nice glaze.
  7. The salmon will flake with a fork when ready.
  8. Eat & Enjoy.

 

Just like a good data center, this dish is sure to be mouthwatering!

CCIE-DC Written In Ten Days? Am I Nuts? Probably.

Well… I am about to embark on a journey to Cisco Live!  Yep.  I set a goal several months ago that I would prepare for an exam that I would take while I was there.  Today, it comes down to the wire — I have ten days till I arrive and I have not had the opportunity to study much — if at all.  So, today I begin my challenge.  Perhaps the hardest one of my life — well other than losing some weight.  Can I?  Is it possible?

 

Can I prepare for, and pass, the written CCIE-Data Center exam in approximately ten days?  Before we take bets, let me preface this with the following caveats and statements:

  • I have extensive data center background.
  • Networking (R/S) drives me nuts.
  • I do not have my CCNA-DC or CCNP-DC.
  • I refuse to use cheats — this is against the rules.
  • I don’t really think I can do it, but damn it man — I am going to give it my damnedest try.
  • I don’t know any speed dealers — I don’t really want to either.

 

So… off to the races.  Sorry sweetie, I might be nose deep in books for a while!