Sep 01, 2012

The CFO was thin-skinned – and short-sighted
The best privacy policies are useless if they’re not enforced
The first breach got a cover-up when it needed fast action
The company ignored the first rule of Digital Forensics
Avoid data storage hoarding.  25 years of data was stored
Management offered mea culpas to the FTC instead of transparency
A lack of accountability for the first breach made a second breach inevitable
A final word

What to do BEFORE The Feds Contact You

“I thought the hacked data center was in Fredericksburg.”  It was Allison Sobine’s third week at Calamatis Hotels and it was already hard to keep track of all the problems.

Arnie Davis looked grim.  “That was last month.  Today they hit Portland.” Allison couldn’t believe her ears. “How? That’s our fourth security breach in three years!”

They raced downstairs to the IT department.  “So… credit card data? Personally identifiable information?”  Arnie gestured helplessly.  “Yes, and yes.  Portland swore they patched the system, but…”

A voice shouted “Allison!”  It was Eileen Collins from legal, waving a thick sheaf of papers, still warm from the printer.  “The FTC just filed suit. Our CEO is already getting calls from the press – what should he say?”

Calamatis Hotels isn’t a real company.  But the issues they’re facing in this case study are very real – and few companies are fully prepared to manage a crisis of this magnitude.

How can something like this happen?

The CFO was thin-skinned – and short-sighted.

The former head of IT had warned the CFO for years about the need to upgrade their out-of-date security.  But her requests for funding grated on the CFO.  She was forced out and replaced with a rising star that wasn’t ready for the job.  The young head of IT got along great with the CFO -- in part because he knew better than to ask for expensive security upgrades.

Action Items:

  • Updating security is cheaper than cleaning up after a breach.  Don’t let systems go unpatched and untested.  Have them audited and penetration tested by an outside company that’s trained to see gaps that insiders will miss.
  • Top management -- not just in IT but in the C-Suite -- must agree on an effective security plan and invest accordingly.
  • Appropriate security policies and procedures should be put in place.  Regular internal and external audits should be conducted to assess whether policies and procedures are being followed.

The best privacy policies are useless if they’re not enforced.

Calamatis’s privacy policy was great – on paper.

But employees can’t uphold a standard if they don’t know it exists.  Calamatis suffered a lot of turnover in IT in the past year.  Few of the new employees had ever seen the privacy policy; almost none of them had any idea who was responsible for compliance.

Action Item:

Every six months, review the privacy policy with staff to ensure everyone knows what is promised and who is responsible for compliance.  Yearly, stage an unscheduled drill to test the company’s reaction time.

The first breach got a cover-up when it needed fast action.

The day the first security breach happened, the IT department applied some quick fixes and hid the problem from upper management.  The irony is that while management didn’t learn about the security holes, the entire hacker world did.  A step-by-step workaround for IT’s hastily-applied patches was published on the web by a Vietnamese hacker, and was the basis for the next three attacks from China, Romania and Indonesia.

Action Items:

Immediately after the first breach:

  • Isolate the system and bring it out of network to preserve both volatile and non-volatile evidence.  If your internal resources are not trained in how to do this in a forensically accepted way, bring in outside experts.
  • Hold an internal investigation if resources are trained and capable, or involve experts, and do so early.  Determine a) what was stolen? b) who stole it? and c) where did the safeguards fail?
  • Engage outside legal counsel with FTC (or similar) experience.  You need someone who understands how the government will view the issue and can offer objective and practical advice.
  • Determine if breach notification is required according to state laws.
  • Bring in a trusted outside consulting firm to do an audit.  Conduct multiple penetration tests.
  • Remediate and fix the damage.
  • Resume operations.

The company ignored the first rule of Digital Forensics

The first rule of Digital Forensics is simple – preservation.  That is, do NOT “step” on the evidence.  Taking quick action in a crisis is important, but taking the wrong actions will only make a bad problem worse.

At Calamatis Hotels, the CEO ordered the company’s internal IT people to go through all the computers involved to figure out what happened.  This was a bad idea.  The IT department worked on the original hard drives, altering items that the FTC considers evidentiary.  They moved suspicious emails to new folders, as well as copied key documents and highlighted potential malware.

From a Digital Forensics point of view, this is like 20 detectives walking through a crime scene putting their fingerprints on everything including the murder weapon, while tracking blood everywhere.

Action Items:

  • Do not conduct an investigation on any electronic storage device (hard drive, flash drive, server, cell phone, etc.) prior to collecting the data in a forensically sound manner.  The data needs to be preserved and collected in a manner that allows for verification.  For example, a forensic image of a hard disk drive involves a bit-by-bit copy being sent to a sterilized destination drive.  This image is backed up and the “working copy” once verified, is used to conduct any subsequent investigation.   Additionally, proper documentation of chain of custody record and the forensic process is a MUST.
  • Upon discovery of an issue (for example a breach or theft), secure the electronic media by preventing access to it.  Continued use of a computer or other device could result in the permanent loss of data of evidentiary value.  There are times when a ruse may be necessary so as not to raise the suspicions of those who may be the target of the investigation.
  • When hiring an outside, independent consultant/vendor, consider carefully the credentials of the people that will collect the data and conduct any future analysis.  Do these credentials include government training and experience?  If the FTC or other government agency gets involved, will they trust the expertise of the people you have chosen?

Avoid data storage hoarding.  25 years of data was stored.

Time and volume of data are two key cost points.  The more data you have, the more it will cost to collect and preserve it.  Make sure you have a legitimate and practical data retention policy in place that serves your business needs and the  regulatory requirements of your industry.

Action Items:

  • Consult legal counsel regarding your industry’s data retention requirements.
  • Only retain the data that is required.  An important caveat:  never delete any data if you are in litigation or are aware of any pending litigation.
  • Once a retention policy is in place, be sure everyone in the company is trained in its policies.
  • Always utilize “law enforcement” trained Digital Forensics professionals to preserve evidence and testify when needed.  Self-collections or collections by e-discovery vendors can create great exposures.

Management offered mea culpas to the FTC instead of transparency

When the FTC called, management offered a set of mea culpas and blamed the breach on a software vendor. The FTC was suspicious, but since Calamatis had a good record to date, they let this pass. But when the second and third breach happened, the FTC knew it could no longer trust Calamatis and action had to be taken.

Action Items:

When the FTC contacts you:

  • Do not ignore the subpoena.  Consider hiring outside counsel that is either former Assistant United States attorneys or that has experience in representing clients being investigated by government agencies.  Listen carefully to their advice.
  • In addition to bringing in experienced outside counsel to rebuild the trust  with the government agency, strongly consider bringing in an independent computer forensic company whose personnel have law enforcement or government experience.  Offer to provide the government agency with the CVs for the key outside people who will be involved.  The experience and background of this outside team will often reassure them that the preservation and investigative steps will be in accordance with standards that would be expected from federal law enforcement.
  • Continue to update them in regular intervals according to their preference.

A lack of accountability for the first breach made a second breach inevitable.

When IT covered up the problem instead of bringing management fully in the loop, management should have immediately communicated that this is unacceptable.  Instead, management chose to participate in the cover-up. There were rumors company-wide that a breach had happened, and that while harsh words had been spoken, management had not held anyone accountable.

This made a second breach inevitable. When the crisis passed, no one worried about what would happen if they were hacked again.

Action Items:

Transparency and accountability are critical in all aspects of business, but especially when you are safeguarding your customers’ data.

Here is what the Calamatis team should have done.

  • The responsibility must start with management.  Appoint a Chief Privacy Officer who is accountable for customer data security and hire a reliable team that supports data security efforts.
  • Involve human resources to consider whether there are grounds for termination of those involved in the cover-up.  Incorporate cultural training that encourages honesty and integrity.  Employees must know it’s important to speak up when something is wrong.
  • Protect your customers’ data as carefully as you would your own intellectual property.

A final word

A crisis like the one Calamatis Hotels faced in this case study is costly, distracting, and ultimately damaging for any business.  You can help prevent them by following the guidelines in this email on your own.  Or, call Capsicum to help you set up a safe, well-maintained operational environment that is customized to your unique business.


How did you land at Capsicum Group?

After spending more than 16 years with the federal government, including six years as an IT forensic analyst at the US Securities and Exchange Commission’s Division of Enforcement, I figured that it was time for a change, to leverage my experience, do more traveling and see the world.  Ultimately, I was looking for an opportunity to interact with a different type of clientele and experience something beyond what the government could offer.

In all honesty, I didn’t really go on any interviews other than the one I set up with Capsicum Group.  What really sold me on Capsicum was that the firm felt like a small boutique rather than a monster organization with multiple living parts.  I also liked that Capsicum afforded me the opportunity to establish strong relationships with a tight-knit internal team, which is critical to successfully working on these cases.

With all of the high-profile cases of recent vintage, including J.P. Morgan and Peregrine, why do we continue to see a proliferation of fraud in financial services?

As Gordon Gekko famously stated, “Greed is, for lack of a better word, good” -- at least that is what a certain segment of the population will always think.  As it pertains to the financial services industry, the proliferation of technology, including iPhones and BlackBerrys, has further emboldened people to think they can hide or manipulate data; nothing could be further from the truth!  The reality is that the abundant use of technology simply increases one’s electronic footprint.  For example, even when one deletes a message from their desktop or mobile device, evidence of that communication typically resides on at least two servers and on the devices of multiple recipients.  Similar to a physical crime scene, digital fingerprints can and typically are recovered by investigators.

Prior to joining Capsicum, you mentioned you were an IT forensic analyst with the SEC.  How have advances in E-Discovery changed the way in which the SEC conducts investigations?

When I first got into the industry in the early 1990s, most of my work at the SEC involved feeding reams of paper evidence into large computer databases from which we analyzed the data.  Today, the majority of discovery is Electronic Discovery, whereby we analyze a tremendous amount of data that resides on mainframes, personal computers and hand-held devices.  Advances and enhancements in computer software have revolutionized the E-Discovery process, simplifying the job and therefore allowing the SEC to become much more efficient.

Based on your undergraduate studies, it seems like you could have easily chosen to study law as opposed to information technology; was a career in law ever an option and what made you choose the path you took?

Good observation. Initially, I was on track to go to law school but that all changed in my junior or senior year when I took a forensics course from a part-time professor who worked in crime scene forensics at the FBI.  That experience compelled me to pursue a Master’s degree in Information Technology from American University after receiving my Bachelor’s degree in Criminal Justice from the University of Maryland at College Park.

In retrospect, that class turned out to be a great choice because what I do today is fun, challenging and sometimes entertaining.

Has an E-Discovery assignment ever resulted in unintended discoveries that were not originally believed to be relevant to an investigation?

Yes.  We were conducting an examination on a workstation of an attorney suspected of fraud and were coming up empty handed.  This situation was further complicated by an uncooperative witness.  As it turns out, the uncooperative witness, a woman, was the secretary of the suspected attorney.  To make a long story short, although we struck out finding instances of fraud on the attorney’s computer, we did unearth compromising photos of the attorney and his secretary.  Needless to say, once the prosecutor presented the secretary with this information, we suddenly had a cooperative witness and were able to successfully move forward with the case.

What do you enjoy most about working at Capsicum?

You’ll find the same answer with everyone at Capsicum; my enjoyment comes from the staff.  We work in an extremely demanding industry; without a good staff, trust or collaboration, the ability to meet critical deadlines would be impossible.  The Capsicum crew is really diverse which has made it easy to get the job done efficiently – much more so than other places I have worked.  We are all focused on a single goal.  It’s a win-win situation.

Capsicum Group, LLC, is a technology consulting company devoted to helping businesses get the most from their technology-related investments. Its practices are focused on various disciplines including: digital investigations and forensics, data recovery, electronic and paper discovery, IT security, risk management and technology delivery. Contact Capsicum today at 1-888-220-3101.