By Insight Editor / 21 Jun 2022
Any organization can be a target for a cyberattack — or collateral damage from an attack targeting someone else. If your organization were hit, would you be able to recover? In this episode of Insight TechTalk, we share a remediation success story and a step-by-step action plan organizations can adopt to ensure a strong, speedy recovery.
To experience this week’s episode, listen on the player above, watch the conversation below, or scroll down to read a complete transcript. You can also subscribe to Insight TechTalk on Apple Podcasts, Pandora, and Spotify.
Audio transcript:
Z
The cyber threat landscape has changed dramatically over the past several years. And any organization can be targeted or simply collateral damage from an attack targeting somebody else. The simple truth is this, any organization or several organizations are unprepared and a result, they can sustain staggering or fatal damage. Don't let your organization suffer the same fate. Hello, my name is Z Tinoco, and I will be your Tech Talk host. And with us today, we do have Ara Carter. Ara Carter is a Director of Pre-sales Engineering and Channels for the Quest Microsoft Platform Management Solutions covering Active Directory, Exchange and SharePoint. He provides expertise and solution insights to Quest partners, helping them achieve successful client engagement. Prior to joining Quest, Carter has been a solution consultant for Microsoft Technologies at Electronic Data Systems, consulting with customers on Active Directory designs, directory migration and client deployment, and of course, server consultation. Let's go ahead and give a warm welcome to Ara Carter, Ara, welcome.
Ara
Thank you, thank you so much, glad to be here.
Z
Fantastic, Ara, well, we definitely have a big topic in store for us today, so I can't wait to learn from your experience. And of course, I imagine you have a lot of stories as well as best practices that you're going to send our away.
Ara
Absolutely, have a few.
Z
Just a little bit.
Ara
Yeah.
Z
All right, well, let's start with the first one, the first question I have for you and a lot of us want to know is that we are aware that the prevalence and disastrous impact that ransomware has on organization, especially over the past several years. We know as well that ransomware is getting more and more sophisticated every time. What are you currently seeing in the way of ransomware attacks? And also to add to that, tell us about what you know about the current threatscape.
Ara
Wow, that's a loaded question. My team we've seen a lot of organizations now being hit, so it's not just a few of the bigger businesses, it's going across multiple sectors now. So small businesses, government agencies, school systems, school districts, healthcare providers, we're seeing lots of lots of different sectors out in the business environment that are being hit today by ransomware. So it seems to be just happening across the board and a lot more often.
Z
Yeah, in other words, I know every organization can be attacked, there's really not one specific area, yeah. Okay, well, it makes me wonder too then is there still organizations and individuals that have this mindset that now that's never going to happen to us, we're fine? (Ara laughing)
Ara
Unfortunately, yes. We still come across customers every day that they have this mindset, they don't think they'll be hit. However, it's best to assume that you're going to be attacked at some point, whether it's down the road or tomorrow. I mean, it's just best to prepare for an attack.
Z
Yeah, I mean, it's just a matter of time. And finally, let's say those individuals just finally give in and say, "Okay, I hear you. I will accept that it can happen or is a possibility that it's going to be there eventually." If you don't mind telling us, is it just as simply as paying for a cybersecurity solution and I'm good or are there actual steps that we need to take to be better prepared?
Ara
Oh, well, first of all paying for a solution is a step in the right direction, but you got to have a plan. So you got to reduce that exposure, then remediate it. So increasing the protections is critical across your systems. You need to ask yourself a few questions, how would you recover these systems if you were under attack. So what would be your first steps? You got to plan for that and think about that. Then where would you start adding additional security protections? What solutions would you add to your processes in order to protect yourself?
Z
I mean, it always starts with the plan, right? I mean, you have to have-
Ara
You got to have a plan.
Z
Got to have a plan and it's not, I mean I wish it was that simple, if it was that simple, we wouldn't be worrying about it. Like I'm going to pay for something and I'm all set, no, you have to always prepare.
Ara
Absolutely.
Z
So, okay, that's good to know. Now let's talk about, I'm assuming you've seen several or been witness to various organizations suffering some sort of ransomware attack. How does this typically unfold? If you can give us a perspective of what typically happens, and then how long does it take for the organization to recover after an attack?
Ara
That's a good question. So if you don't have a solution or you don't have the backups in place we're seeing an average downtime of about 21 days for a ransomware attack. Some customers are actually reporting longer downtimes so it can be quite lengthy depending on what you have in place. Now if you are a Quest customer, then let me tell you about a real attack that we've helped out with. So we had a customer that was in the global manufacturing industry. They were hit by ransomware. It impacted their 17 domain controllers which were spread across multiple continents. It also even scrambled their passwords for 98% of their user accounts. So it was quite a bit of a bad situation that they were in.
Z
Yeah, it definitely sounds very serious. So what happened?
Ara
Well, luckily they owned Quest, so they owned our recovery manager for Active Directory Disaster Recovery Edition. So using the solution they were able to perform a phased recovery. So it allowed them to prioritize the domain controllers they needed to get up and running quickly so they could restore access to their environment and restore their business so they can get back up and running. Following this approach they were able to get five of their prioritized DCs up and running within about two hours. They were even able to reset those passwords. They were so happy about the time savings and having this solution in place that the project manager told us there was no way they would've recovered this quickly had they not had the Quest tool in place.
Z
I mean, I don't know about all of you but I love a happy ending, a success story. But again, with every success story, there's also some stories that are not so successful, now the thing is learning from them, right? Is being to understand, "Hey, what could I have done differently" or also learning from other successes and duplicating that. So thank you for sharing that.
Ara
Yeah.
Z
All right, well, let's move on to this next section here. And what are some important lessons any organization should consider when they're thinking about creating their ransomware strategy?
Ara
Man, there's so many lessons out there, and we don't have enough time to cover 'em all, but-
Z
I mean, we have two hours, so let's just make the best out of it.
Ara
Well, the first thing being, you got to air gap your backups. So your backups are going to be the most important thing to have in place for doing a restore. These backups, having 'em in place, it's important but they got to be at air gaps so that they're not corrupted. I mean, how are you going to severely bring back your environment if your backups are no good? So that's got to be the first thing is having a good system of air gaping those backups. And luckily our Recovery Manager Disaster Recovery Edition, it includes secure storage server. So we provide a way of checking those backups, and making sure that they're virtually not accessible to the hackers in the first place.
Z
I mean, I think about, yeah, I would be frustrated if we get a backup and it's outdated, it's like, wait a minute, when was the last time we looked at this? And make sure that it's relevant and it's going to be like business as usual. So that's very important.
Ara
Yeah, next you got to have a plan for your attack. So you got to sit down and really develop out what's going to be my plan in case I do get attacked. And this is going to be putting in place a virtual war room where you have all of your teams together, so the backup team, storage team, network team your server team, security, all the different parties that are going to be involved in doing a restore in case you get attacked, you got to have this virtual room in place and ready to start going down the list of actions you need to take. And then you can't just limit your plan to Active Directory recovery. So you got to also take into consideration the other impacts that are there such as to your network, your VPNs, your server security hardening, your endpoint detection. So you need to have all of this in place and considered in your plan as well. And it also got to make sure this plan is accessible. So if you do get locked out, you get ransomware hit, you may not have access to the servers where your plan documents are stored. So you want to make sure that either these plans are printed out and accessible by everybody on your team or place 'em on a cloud storage device such as Dropbox, so that you can get to 'em even if you are in a ransomware situation. Finally, you got to test that plan out over and over, constantly, to make sure that your plan is valid and that you can recover in cases of crisis such as ransomware.
Z
Wow, okay, when you said plan, you really meant plan.
Ara
Yes, you got to have that.
Z
So it's a multiple step process, but again, it's making sure that, I mean, just a recap, I heard you say, well, create a war room, get everybody involved that needs to be involved, make sure that your plan covers more than just AD recovery, make it accessible so that way everybody knows where it's at or where they need to go, test it, prepare a phase of recovery approach. Okay, makes sense, but is there anything else?
Ara
Well, it's not just about speed. So you want to make sure that you get things done and don't get reinfected. So you want to make sure that you have flexibility in your recovery approach and that's whether you're going to use clean OS recovery or bare metal recovery, you want to make sure you choose the right recovery that's based on your situation. So if you're going to use bare metal, you got to remember that the target servers have to have the same physical disc layout. And this is also going to include some components that are really not needed for Active Directory restoration. And so this gives ransomware a place to hide in some of these other areas, some of these other disc areas if you choose to use bare metal recovery, whereas if you use a clean OS, this is going to reduce some of that risk because it only includes the information that's necessary for the Active Directory recovery. And additionally, you can restore it to a virtual machine using this type of restore option and then Recovery Manager for Active Directory Disaster Recovery Edition is going to be doing malware checks on the backups before you do the restore. So we're going to be checking all the way around depending on what option you choose to use in restoring your environment.
Z
I mean, I know you put me at ease when you said, I mean, all this work for you to get reinfected again, would be a nightmare, right? It's like you're constantly fighting a battle that it never seems to end. So it's so important that you take these steps so it doesn't happen again.
Ara
Absolutely.
Z
So in that same note, what are some common misconceptions or let's just say mistakes with backup and recovery?
Ara
Man, so some of the customers we talk to, they think that their native backups, their native Active Directory backups is enough, but native backups, they're going to have your system state or even bare metal backups. They can contain areas where you could be exploited, again, by that same ware attack to begin with. So aside from using that bare metal recovery native tools can be extremely difficult to walk through that 40 step process. Scripting and homegrown solutions is another thing we hear about. We see a lot of customers using homegrown ways or homegrown software in order to do their recovery. And this is only going to be as good as who developed it. There are some third party backups out there similar to ours, but you got to make sure that it does not only just restoring of the deleted objects but also can do forest recovery with automation and help you in doing the reconfiguration as you walk through the full forest, full environmental recovery of your Active Directory environment there. So it's quite a few misconceptions out there based on what we're hearing and talking to some of our customer base.
Z
Yeah, definitely, quite a few for sure. Now you mentioned a few solutions so far, but if you don't mind, walk us through the top reasons for choosing the solution like you mentioned, RMAD DRE. I use the acronym by the way, I couldn't say the whole thing.
Ara
Yeah, use an acronym. I'll go ahead and say the full name. So Recovery Manager Active Directory Disaster Recovery Edition is a solution that we have that's specifically for restoring after ransom were hit. So it provides quite a bit of features and functionalities from object level and attribute level protection. So if you have deleted objects or deleted attributes, this solution is going to take care of that, bringing those back and these going to be accidental changes, inadvertent changes to your environment, which administrators make every day. You want to have a way of recovering from that that's not going to be a lot of time and effort. Group policy object protection, so we can recover changes that are made to group policy objects as well. So just like object and attribute level recovery, we also talk about and cover GPO recovery. So having those GPOs back in case someone made a change to a GPO, maybe it was part of the ransomware that GPO got changed, we're able to restore that from backup. In situations where you don't know what's changed, it's going to be good to have a way of reporting on that. So this solution comes with comparison reports. So if you're not aware of what's changed in your environment, we have a solution within recovery manager that looks at the backup and looks at the live environment and then reports on the differences. So you can quickly see, "Okay, these are the changes that have been made to my environment and I can go in and quickly restore these changes."
So those are the first ones, we have a few more, so you want to have a system that does automated forest level recovery, and that's really the root of our recovery manager solution. So we can walk you through all of those steps that Microsoft is saying you have to cover, we automate that for you and do the full forest recovery without having to go out and touch each domain controller or do any scripting, we can do that automatically through our solution. We also offer some flexible recovery options. So you have a way of recovering what's best for your environment or the approach that's going to follow what's best for your environment. And then finally to make sure those backups are protected and secure, we offer secure backup storage. So we give you a way of storing your backups so that ransomware can't infect these backups and they're available to you when you're ready to use them. So we secure server to make sure that you're able to perform clean backups and clean restores whenever necessary. All of those features are provided by a single solution, our RMAD DRE tool.
Z
Wow, okay, well, those are amazing, thank you for sharing those top reasons for choosing those solutions. Now, if you want more information or if you want to learn more about what we just discussed, don't forget to look for Quest solutions on our page on insight.com. We love for you to engage with us. And of course would like to have a conversation with you. With that Ara, thank you so much for taking the time to speak with us and share your knowledge and expertise. And of course we'll be hopefully seeing you around.
Ara
All right, thanks for having me.
Z
All right, thank you very much. And with that being said, till next time, everybody, thank you for another successful Tech Talk, bye-bye.
Diversity, Leadership & OD Manager, Insight
Z Tinoco is a diversity, leadership & organization development manager who believes in building teams, inspiring minds and creating authentic connections. He helps people reach their goals and find success through humor, leadership and a diverse mindset.
Presales Engineering Director, Quest Software
Ara is a director of presales engineering and channels for Quest’s Microsoft platform management solutions, covering Active Directory, Exchange and SharePoint. He provides expertise and solution insight to Quest partners, helping them achieve successful client engagements. Prior to joining Quest, Carter was a solutions consultant for Microsoft technologies at Electronic Data Systems (EDS), consulting with customers on Active Directory designs, directory migrations, client deployments and server consolidations.
Receive automatic notifications when new episodes are released.
