Please let us know if you’d like to see more or specific tutorials.

“Essentially unchanged from last year, only 21 percent of
organizations were fully compliant at the time of their Initial
Report on Compliance (IROC). This is interesting, since most
were validated to be in compliance during their prior assessment.
What causes this erosion over the course of the year?”

….

Organizations struggled most with the following PCI
requirements: 3 (protect stored cardholder data), 10 (track and
monitor access), 11 (regularly test systems and processes), and
12 (maintain security policies).”

Here’s another passage that surprises no one, from pg. 29:

“the secret to maintaining compliance lies largely in treating it as a daily part of conducting business. To achieve this, the correct mind-set must be instilled across the organization, and this type of integration must come from the top down.”

I may be projecting but while I read the report, I kept visualizing the ops and security teams scrambling between the initial and final assessments to hurry up and get compliant. Given the data above, deep down they knew they’ll be doing the same thing next year. This got me wondering: how much time, money, and opportunity cost is spent scrambling to get compliant? How can we reduce this cost?

Here’s a hypothesis: the cost of running a compliant shop is less than the cost to run annual fire drills. Boy I’d love to test this hypothesis. This also fits into my favorite question about metrics: is control effectiveness increased simply by the fact it’s being measured? Mind you I’m talking about measuring, not auditing. This PCI report clearly shows audits don’t improve controls throughout the year. What if we applied metrics to key operating controls under the PCI scope? Would a minor investment in spinning up a metrics program be less than the annual effort to get compliant? Is the cost of maintaining a control less than the cost of scrambling to fix it?

I took the liberty to identify a candidate set of metrics that may help answer these questions. I ran through the requirements and selected 23 metrics. I don’t think we need a metric for every requirement. My goal was to identify metrics that provide visibility to a group of controls. E.g. % of users with completed role verification per some schedule. This metric may knock off multiple requirements e.g. least privilege, segregation of duties, terminations, vendor, and remote access. I also focused on the operational controls vs. design decisions e.g. metrics aren’t a fit to measure your encryption design.

Here are the 23 (feel free to scroll down and read them later if you want more monologue):

• An interesting observation is that our MoneySec list of metrics only contains 15. Recall the MoneyBall inspired list of metrics is a candidate list to identify key controls that correlate with reducing incidents. If all our hypotheses were correct, we could show that compliance focuses on too many things that don’t matter (but that’s another post).

I fully understand you’re not going to spin up a metrics program for all 23 without some demonstration of value. So I propose you start with one metric, my favorite, that covers many PCI requirements: number of scanner-sourced vulnerabilities past due. This is a great metric because it requires:

• you regularly scan
• have a process to rank vulnerabilities
• have a pre-negotiated service level in days to fix vulns based on severity level
• have the ability to age vulnerabilities

This metric knocks off all kinds of PCI reqs e.g. conducting scans, mitigating vulns, default creds, patches, config vulns, everything a scanner catches that’s listed in PCI.

Here’s an example using our Vuln Tracker tool to age vulns from popular scanners. Note you can use other tools or a spreadsheet (as Brian Keefer demonstrated in our MoneySec presentation).

This example shows a couple hundred vulns overdue, some older than 3 months. From a metrics point of view, you could set a target value for Overdue Vulns at say 10 vulns per month. Here’s how that might look using our Metrics Manager (again, any old spreadsheet will suffice).

This shows the history of mitigation performance and how well we performed to our expected target.

What do you think? Does the simple fact of looking at something change its behavior? In my experience, yes. So if you have the annual privilege of meeting your QSA, see if you can save your company money by maintaining compliance vs. getting compliant on an annual basis.

Please do provide feedback on the list of metrics above. It was just me and my pint who produced them and I’m sure they can improve!

Simple. No free lunch.

Internal consulting, project support, business enablement, whatever you want to call it, is a service. This service should be recognized, advertised, resourced, and measured. At one point in my history, we put engagements into time buckets e.g. < 4 hours, < 1 week, 1-3 weeks. Each security pro then had a certain % of their time assigned to internal consulting. When they became overwhelmed, it was their job to escalate to me. We had a published process to assign, gauge, and prioritize the work effort. During a time of layoffs and cost cutting, we actually justified and earned another FTE because we could show the demand and value. I said we either start saying “we’ll have to get to you in xx days” or we need more quality folks. Magic.

Lots of anecdotes to share but I’ll spare you. I will leave you with some of the tools we used to formalize the process. Mind you, not everyone on the team supported this formality. Some folks liked being the Go-To-Guy. They didn’t appreciate having to say, “I’d love to help you directly but we have a standard way to serve you, please visit <portal url> and enter your request. We’ll get back to you within 1 day.” The magic happened when the go-to folks were totally swamped. They got to remain focused and the business was served.

In a bit of self-promotion, we recently added a template in Risk Communicator to support general consulting requests. This template is a bit different because it contains a set of questions to identify risks, similar to control based templates like PCI. The goal is to provide a consistent set of questions enabling assessors to quickly understand the solution and key control requirements. Once you define a well formed risk statement (impact and likelihood evidence), you have the option of completing the workflow to help the business improve their decisions where to spend.

This approach isn’t a substitute for a comprehensive assessment, it’s a quick hit and can/should be customized. Here are the base questions in the Risk Communicator template: (apologies but I can’t convince wordpress to get the table in html, here’s the pdf)

basic_questions

Of course having a slick risk assessment application pales in comparison to a well defined process. Here are a few generic deliverables to get you started:

Project Support Overview Slide

Project Support Workflow

Simple RACI for Project Support roles

Some of these may be too basic or even complex for your group. Recall that one person may play multiple roles.

Few more notes:

• A great way to get started is to simply formalize how the security team is engaged. An internal portal is best e.g. custom sharepoint site, but a shared mailbox will do at the start.
• Be sure to enforce the process. Ignoring the engagement point defeats the purpose.
• Get exec support and market your service. The engagement point, process, and key SME’s should be advertised like rock stars.
• Project support should be recognized in the list of services your team provides, whether or not you have a fancy service catalog.
• Metrics: how about

• % requests served within SLA
• # of requests +/- predicted per quarter e.g. we anticipated 30 but received 60!

• Get credit. Internal consulting should be included in your perf goals and review. cha-ching.

The ultimate goal is to serve the organization while enjoying your job.

I hope you find these useful. Please contact me or leave a comment with feedback or questions.

Today we’re seeing GRC and other vendors consume vuln scans like they’re commodity pork bellies.  They add on some workflow, connectors, reporting, and presto, you have the technology capable of supporting a great assessment service (almost as good as bacon).

This post has two purposes. One is to wax all nostalgic on how much I love vuln scanners. The other is to help explain why Third Defense just released a reporting extension on top of them. No, we are not providing all the workflow like the GRC crowd. We’re simply responding to a customer request who asked if we could help them out. Since I’ve been asking vuln scan vendors to produce this report for years, I said YES.

The report is simply to show vulnerability age compared to policy. It’s easy to age a vulnerability on specific hosts across scans. It’s also incredibly powerful for a security team to report on overdue vulns per asset group. The goal is to increase accountability and drive risk acceptance|mitigation decisions across business owners. Odds are you already have pre-negotiated mitigation time frames per sev level with the Ops group. Some folks include these time frames in policy. Others yet call them Service Level Agreements.

Either way, you now have an easy way to show vuln age. Here are the steps:

1. Paste your existing asset group names, owners, and IP ranges into the Vuln Tracker app
2. Assign a remediation date per group and sev level
3. Upload scans

The result is a simple histogram that can show:

• Age of all vulns across all groups
• All Overdue vulns
• Vuln age per group
• Overdue vulns per group

Here’s a screen snip (click to expand):

I fully expect vuln scan vendors to add in this report someday. It would be cool if we helped hasten the process. Until then, I encourage you to try out this report. The increased visibility into remediation performance works wonders. In my experience, maturing the vuln mngt process is the easiest across all of security. It’s rewarding to sit down with ops every month and translate vuln definitions into risk statements for your business. Take the next step and verify the appropriate remediation occurs.

As always, we welcome your feedback, even if it’s to remind us that we’re crazy to add a feature like this :)

Quick note: out of the gate we only support nessus. Let us know if you’d like to see more.

Thanks to Justin Somaini for his pointer to the University of CA ERM program. Yesterday I had Grace Cricket’s webinar on ERM running in the background. When something caught my ear, I’d check it out on UC’s fantastic Resources site. Below are some notes I jotted down. Huge disclaimer: I didn’t take the time to listen to everything or view all the resources. The purpose of the this post and these notes are to incite interest and Thank Grace and co for sharing!

Begin rough notes and soundbites:

“How do you know if you’re doing well?” Develop KPI’s, find the data. Instead of asking (stakeholders) what keeps them up at night. Ask them how they measure success. (me: Brilliant!). In the past, data was ad hoc and manual, not repeatable. Need technology to manage information. Selected IBM to develop solution.

- Developed ERM maturity model. Use S&P rating methodology.

- Retrospective reviews of impacts > $50k. (me: great source of evidence)

- ERM is basically a COE to cross pillars of risk management.

- developed an ERMIS: single portal, organizes information for monitoring performance.

- Developed Risk Assessment workbooks. (me: these are simple but a great starting point. I fully expect their use isn’t consistent, not always data driven, subject to politics, etc. but it’s a start for them)

- Probably could have spend a lot of money on fancy tools but folks are able to use simple tools, empower them to use something familiar e.g. xls

- combination of pull surveys vs. push assessments (me: the xls workbooks). complement by ERM maturity rating e.g. rims.org? 107 ERM activities across 5 categories. Self rate cmm like scale to rate ERM program itself.

- question from audience: what’s the motivation to start: 1. need an overall champion e.g. CRO. 2. need for a unified response to catastrophic incidents

- (52 mins in) ERM tracking: identified ~40 Ent. Risks across broad categories e.g. the one IT risk entry: Title “Decentralization of systems leading to data inconsistencies and fragmentation.
Mitigation: Senior leadership has recently put in place storage contols in this area;
Development and Maintenance Standards and (hard to hear?) local policies.
Data, Monitoring & Reporting: Reported at local level; Programing quality assurance and testing: approvals by programming managers and users before moving new systems or changes to production.

- advice: identify lowest hanging fruit, iterate

- about 700 KPI’s across ERM (me: I assume across the whole UC ecosystem)

- Risk Appetite: to begin, just show performance against average of institutions across UC system e.g. yellow is 5% of average.

- question from audience: Do you charge each institution for ERM services?  No, funded centrally.

- Can ERM quantify it’s benefit e.g. “elminimated cost of claims system @ $4M, ERM costs 2.5M/year. Improved credit rating.

- 1:08 in: First big win was workers comp. “cost of risk for fy09/10 reduced from $18.46 to $14.76″ ??

- first win was leveraging individual risk assessment tools. Next win will probably be to leverage the portal roll-up.

- How is CRO perceived e.g. auditors? Depends. Use qualitative to get past “sin factor.” Need to leverage qual and quant. Not an audit, no overlap from compliance. Focused on helping individual owners manage risk more effectively.

- CRO is a generalist to break down silos e.g. Financial, Safety, Compliance, IT Governance.

End rough notes and soundbites.

I don’t know if UC’s ERM program is good or bad. I do know I appreciate the insight. I also know we need more sharing. Please let me know as you come across great examples we can all learn from.

Thanks again Grace!

Anyway, here’s the official slideshare link to my session on How To Find The Right Amount of Security Spend.

Update: 30 min video here.

Here are the slides in .pdf:  How_Much_Security. Let me know if you’d like the source ppt slides (do you feel less comfortable downloading a .pdf or .ppt :).

Aside: one of my favorite talks was from Myles Conley http://www.sourceconference.com/seattle/speakers_2011.asp#mconley

Keep on the lookout for his slides when they’re available. Myles did an excellent job analyzing breach data to emphasize what roles infosec organizations really need i.e. web app dev’s may be cool but perhaps project managers may be more useful…

The first is to recognize the resistance when building or expanding a metrics program. I haven’t heard anyone say they don’t want to measure. What I do hear is they don’t have enough time. Taking time away from business as usual and committed initiatives is tough so executive support is required. Do your executives really want to understand acceptable risk and efficiencies or not? If so, read more.

The second is the simple tactic of requiring each metric to have a target value. These targets define what “acceptable” means to asset and control owners. Targets are also a good forcing function to make sure a metric passes the “so what” test. Defining target values also normalizes % and integer based metrics. Stakeholders don’t have to understand every detail but they can quickly see if performance is above/below a committed level.

Now back to the inspiration for this post. Below is a list of metrics derived from a couple IT shops of my past and what I’ve seen/heard across my journey. You’ve seen me organize metrics by IT services and technical domains. This snapshot leverages the questions to define security investment: are we operating at acceptable risk, and are we as efficient a possible. I like this more business-friendly structure and for the true crazy elite, it aligns nicely into a balanced scorecard. Obviously I can’t include target values for you but please make sure you do.

Are we operating at acceptable risk?

Applications

• % internal projects using SDL
• % external projects with SDL in contract
• # pre-production (final security review) sev 1 security bugs
• # post-prod bugs for new/updated apps
• # security bugs identified in production application assessments
• % SDLC staff trained

Data

• % of PII encrypted/obfuscated per policy
• # of PII policy violations identified with DLP

IAM

• % time to complete User Access Reviews
• % of administrator accounts validated per policy cycle e.g. quarterly
• % user accounts managed through automation (or central directory)
• % of employees and non-employee workers with baseline access profiles
• % accounts deprovisioned per SLA

Vendor

• % critical vendors e.g. with PII, assessed per policy
• # of outstanding vendor risks

Device (aka scanners are your friend)

• % workstations/laptops outside patch SLA (e.g. critical 7 days, important 30 days, maintenance 90 days)
• % workstations/laptops with vulns outside SLA
• % workstations and laptops Managed for Security where “managed” is a bundle of healthy agents e.g. AV, HIDS.
• same as above for servers
• %  accuracy of cmdb vs network enumeration (scan)

Network

• # Network attached devices with vulns outside SLA
• % Accuracy of inventory (scan enumeration vs. cmdb)

Security Monitoring

• % network monitored per policy
• % servers monitored for security (may add a separate item for DB’s)
• % security tickets resolved within SLA

Incident

• # of critical incidents
• # of moderate incidents
• % incidents managed per SLA (identification, declaration, QFE, root cause, resolution)

People

• % staff trained per policy
• -Include outreach programs as appropriate e.g. cross-crash team meetings, brown bags, newsletters, sponsored socials, etc.
• Might be cool to break out social engineering incidents separately?
• # compliments showered on security team (wouldn’t that be nice)

Are we as efficient as possible?

Security Team Improvement

• % security tools and deployment rationalized (gauge of architecture discipline and utilization)
• # of security innovation ideas transformed into action (to encourage team participation) (yes, set a target value here too)

Security Program

• % Infosec spend aligned to strategy (which is/should be aligned to biz strategy)
• % security processes documented with assigned SLA’s (documented means RACI, swimlane, even SIPOC for all you nerds)
• % of key security processes with completed FMEA (failure mode exception analysis i.e. how can/do we break)
• % security initiatives completed on time
• % security initiatives completed on budget

Business Engagement

• % Line of Business security assessments completed on schedule e.g. semi-annual (aligned to budget cycle)
• % risks identified without status resolution (accepted, mitigated, mitigating). This focuses on risks identified during assessments and consulting e.g. evaluating cloud services, authentication requirements, etc.
• % consulting projects complete within SLA (volume and quality of LoB and IT projects supported with internal consulting)
• % capacity of internal consulting team (gauge supply/demand)

Compliance

• # “significant” audit findings identified
• % accepted remediation actions completed on time
• % audits following process (pre-scheduled, proper scope, duration)
• security violations managed per policy aka exception duration and risk acceptance by asset owner

…………..

Phew, looks like a lot of work and it is. If you really want to go crazy, create a performance roll-up for each section or the list as a whole. Target values normalize metrics so you can communicate % above/below expected progress. For perspective, it took us years working through the above and unfortunately I didn’t always get to see it through i.e. bank collapses :-) Plus, if your experience is like mine, your team will resist most of the above because they don’t have the time. It’s up to leadership to set the work priorities, reward improvement, dissolve heroism, and recognize the costs and benefits of metrics. In my experience, the start-up pain is well worth the reward. Not only will you have a solid foundation to measure, you’ll have a great source of evidence for your risk assessment process!

I encourage you to critique, add, subtract from the above. I’ll update the based on feedback and maybe crowd source this a bit. There’s no one right answer and metrics will evolve over time. Hope this was helpful!