SIRA Webinars

Quick cross post if you’re interested in watching some of the SIRA webinars. They’re available to everyone on the SIRA blog.

Keefer/Pfost’s presentation on “Moneysec: Applying Moneyball…” here: https://www.societyinforisk.org/content/sira-monthly-webinar-982011-17-gmt12-edt9-pdt-brian-keefer-and-jared-pfost-moneysec#comment-3

Herath’s presentation on Cyberinsurance here:

https://www.societyinforisk.org/content/sira-monthly-webinar-5122011-12-pm10-am-edtpdt-it-risk-privacy-and-other-legal-concerns

Rosenquist’s presentation on TARA here:

https://www.societyinforisk.org/content/sira-monthly-webinar-8112011-1200pm-edt900am-pst-matthew-rosenquist-tara

Our New Videos!

Hello all. We just posted updated screen casts of all our apps on our web site. Vids are sprinkled throughout and cataloged on their own page. I tried to keep them short so lots of cool stuff left on the cutting room floor. They’re also in HD so you should be able to see the apps this time. Fortunately I didn’t include video of me talking so you’re safe.

Please let us know if you’d like to see more or specific tutorials.

Guest Post: Blue Hat Blog

Quick pointer to a guest entry on the Blue Hat Blog. I’ll post a pointer if my recorded session is available online.

Podcast: hanging with Santarcangelo et al. to transform awareness

Quick post to share a podcast on how to make awareness programs make a difference. Many thanks to Michael Santarcangelo, the Security Catalyst, for letting me join in. It’s good to see there’s room for a process junkie when great practitioners get together. It was also nice to meet the Focus folks who run a well oiled shop. You can find the podcast here. All feedback welcome!

Can metrics save money on PCI compliance?

I continue to be impressed by the VZ team. Their latest PCI Compliance Report continues their contribution of data sharing with the industry. Here are a couple cherry picked passages from the exec sum:

“Essentially unchanged from last year, only 21 percent of
organizations were fully compliant at the time of their Initial
Report on Compliance (IROC). This is interesting, since most
were validated to be in compliance during their prior assessment.
What causes this erosion over the course of the year?”

….

Organizations struggled most with the following PCI
requirements: 3 (protect stored cardholder data), 10 (track and
monitor access), 11 (regularly test systems and processes), and
12 (maintain security policies).”

Here’s another passage that surprises no one, from pg. 29:

“the secret to maintaining compliance lies largely in treating it as a daily part of conducting business. To achieve this, the correct mind-set must be instilled across the organization, and this type of integration must come from the top down.”

I may be projecting but while I read the report, I kept visualizing the ops and security teams scrambling between the initial and final assessments to hurry up and get compliant. Given the data above, deep down they knew they’ll be doing the same thing next year. This got me wondering: how much time, money, and opportunity cost is spent scrambling to get compliant? How can we reduce this cost?

Here’s a hypothesis: the cost of running a compliant shop is less than the cost to run annual fire drills. Boy I’d love to test this hypothesis. This also fits into my favorite question about metrics: is control effectiveness increased simply by the fact it’s being measured? Mind you I’m talking about measuring, not auditing. This PCI report clearly shows audits don’t improve controls throughout the year. What if we applied metrics to key operating controls under the PCI scope? Would a minor investment in spinning up a metrics program be less than the annual effort to get compliant? Is the cost of maintaining a control less than the cost of scrambling to fix it?

I took the liberty to identify a candidate set of metrics that may help answer these questions. I ran through the requirements and selected 23 metrics. I don’t think we need a metric for every requirement. My goal was to identify metrics that provide visibility to a group of controls. E.g. % of users with completed role verification per some schedule. This metric may knock off multiple requirements e.g. least privilege, segregation of duties, terminations, vendor, and remote access. I also focused on the operational controls vs. design decisions e.g. metrics aren’t a fit to measure your encryption design.

Here are the 23 (feel free to scroll down and read them later if you want more monologue):

Metric Title	Category	Description
Post-prod. Vulns.	Application	# Post-production applications vulnerabilities Target: 0 Support PCI requirement 6.5.
Secure Development	Application	% of in scope application development projects that follow secure development practices. Focus on PCI requirement 6.3 and 6.5.
Change Control Security	Change Control	Number of changes that reduce or violate security policy. Focus on PCI requirement 6.4.
Production Access	Change Control	Number of non-operations personnel with access to production systems. Focus on developer access to address PCI 6.4.
PCI Servers	Data	Number of actual in scope servers holding PCI data compared to known target number. Goal is to set a target and verify at <regular intervals>.
Retention	Data	Number of in scope records past data retention policy. Frequency: quarterly Target: 0
Device Management	Device	% of devices managed per PCI requirements. Scanners and/or configuration management agents are good sources to produce this metric. e.g. 1.4 Install personal firewall software on any mobile and/or employee-owned computers 2.2 Develop configuration standards for all system components. 5.1 Antivirus use and management.
Device Vulns	Device	Number of unaccepted patch & config vulnerabilities beyond predetermined time frames e.g. sev 5 within 30 days. target: 0 source: scanner frequency: as frequent as feasible for your org. prerequisites: policy stating patch SLA in days per vuln severity level, identified asset group by business importance. Focus on PCI requirements 6.1, 6.2, 11.2.
Intrusion Detection	Device	% of in scope system traffic monitored by IDS/IPS on a <quarterly> basis. target: 100% Focus on PCI requirement 11.4.
Integrity Monitoring	Device	% of in scope systems with file integrity monitoring on a <quarterly> basis. target: 100% Focus on PCI requirement 11.5.
Firewall Review	Firewall	% of firewall and router rule sets reviewed <at least every 6 months> target: 100% Supports PCI requirement 1.1 “review firewall and router rule sets at least every six months.”
Default Credentials	IAM	# servers/infrastructure devices with default credentails target: 0 source: scanner, manual A&P frequency: <depends on org e.g. quarterly> Focus on PCI requirement 8.
Terminations	IAM	# of Employee terminations outside predetermined time frames Target: 0 Focus on PCI requirement 7.
Role Verification	IAM	% of users with completed role verification per schedule target: 100% source: manual or automated frequency: semi-annual Focus on PCI requirement 7.
Credential Strength	IAM	% servers/infrastructure devices with 2-factor or password complexity reqs target: 100% source: config review, manual A&P frequency: semi-annual Focus on PCI requirement 8.
Incident Response	Incident Response	Number of incidents not handled in accordance to documented incident response procedures. target: 0 Focus on PCI requirement 12.5 and 12.9.
System Monitoring	Monitoring	% of in scope systems monitored per PCI requirements. target: 100%
Unauthorized WLAN	Monitoring	Number of unauthorized WLANs identified on a quarterly basis. target: 0 Focus on PCI requirement 11.1.
Visitor Badge Return	Physical Security	Percentage of visitor badges returned per <time period>. Focus on PCI requirement 9. The goal is to identify an inexpensive metric that provides some attention to physical access. The assumption is measurement in one area will increase the effectiveness of others.
Policy Review	Policy	% policies reviewed on an <annual basis>. Focus on PCI requirement 12.1.
Closed Risk Dispositions	Risk Assessment	Number of risks identified during the annual assessment without a risk owner and disposition e.g. accept, mitigate, transfer. target: 0 Focus on PCI requirement 12.2.
Vendor Remediation	Vendor	# of vendor security findings that have not been addressed within committed time frames. Helps support PCI 2.4 Shared hosting providers must protect each entity’s hosted environment and cardholder data.
Vendor Assessments	Vendor	Percent of vendors who receive security assessments within policy e.g. vendors with sensitive data/services must be reviewed semi-annually. Helps support PCI 2.4 and 12.8. Shared hosting providers must protect each entity’s hosted environment and cardholder data.

• An interesting observation is that our MoneySec list of metrics only contains 15. Recall the MoneyBall inspired list of metrics is a candidate list to identify key controls that correlate with reducing incidents. If all our hypotheses were correct, we could show that compliance focuses on too many things that don’t matter (but that’s another post).

I fully understand you’re not going to spin up a metrics program for all 23 without some demonstration of value. So I propose you start with one metric, my favorite, that covers many PCI requirements: number of scanner-sourced vulnerabilities past due. This is a great metric because it requires:

• you regularly scan
• have a process to rank vulnerabilities
• have a pre-negotiated service level in days to fix vulns based on severity level
• have the ability to age vulnerabilities

This metric knocks off all kinds of PCI reqs e.g. conducting scans, mitigating vulns, default creds, patches, config vulns, everything a scanner catches that’s listed in PCI.

Here’s an example using our Vuln Tracker tool to age vulns from popular scanners. Note you can use other tools or a spreadsheet (as Brian Keefer demonstrated in our MoneySec presentation).

This example shows a couple hundred vulns overdue, some older than 3 months. From a metrics point of view, you could set a target value for Overdue Vulns at say 10 vulns per month. Here’s how that might look using our Metrics Manager (again, any old spreadsheet will suffice).

This shows the history of mitigation performance and how well we performed to our expected target.

What do you think? Does the simple fact of looking at something change its behavior? In my experience, yes. So if you have the annual privilege of meeting your QSA, see if you can save your company money by maintaining compliance vs. getting compliant on an annual basis.

Please do provide feedback on the list of metrics above. It was just me and my pint who produced them and I’m sure they can improve!

In touch with your inner consultant

While helping folks build fun, cool processes like assessing risks with fancy web apps, a nagging trend emerged. Security pro’s are often overwhelmed with random requests to provide advice or approve designs to support internal projects. Some of these requests come early to support official projects e.g. we need your help designing the next ERP implementation. However most are ad hoc, random, and disrupt an already over-crowded work week. Or worse yet, the security team isn’t engaged until the day before go-live! How can under resourced security pro’s get in front of all this?

Simple. No free lunch.

Internal consulting, project support, business enablement, whatever you want to call it, is a service. This service should be recognized, advertised, resourced, and measured. At one point in my history, we put engagements into time buckets e.g. < 4 hours, < 1 week, 1-3 weeks. Each security pro then had a certain % of their time assigned to internal consulting. When they became overwhelmed, it was their job to escalate to me. We had a published process to assign, gauge, and prioritize the work effort. During a time of layoffs and cost cutting, we actually justified and earned another FTE because we could show the demand and value. I said we either start saying “we’ll have to get to you in xx days” or we need more quality folks. Magic.

Lots of anecdotes to share but I’ll spare you. I will leave you with some of the tools we used to formalize the process. Mind you, not everyone on the team supported this formality. Some folks liked being the Go-To-Guy. They didn’t appreciate having to say, “I’d love to help you directly but we have a standard way to serve you, please visit <portal url> and enter your request. We’ll get back to you within 1 day.” The magic happened when the go-to folks were totally swamped. They got to remain focused and the business was served.

In a bit of self-promotion, we recently added a template in Risk Communicator to support general consulting requests. This template is a bit different because it contains a set of questions to identify risks, similar to control based templates like PCI. The goal is to provide a consistent set of questions enabling assessors to quickly understand the solution and key control requirements. Once you define a well formed risk statement (impact and likelihood evidence), you have the option of completing the workflow to help the business improve their decisions where to spend.

This approach isn’t a substitute for a comprehensive assessment, it’s a quick hit and can/should be customized. Here are the base questions in the Risk Communicator template: (apologies but I can’t convince wordpress to get the table in html, here’s the pdf)

basic_questions

Of course having a slick risk assessment application pales in comparison to a well defined process. Here are a few generic deliverables to get you started:

Project Support Overview Slide

Project Support Workflow

Simple RACI for Project Support roles

Some of these may be too basic or even complex for your group. Recall that one person may play multiple roles.

Few more notes:

• A great way to get started is to simply formalize how the security team is engaged. An internal portal is best e.g. custom sharepoint site, but a shared mailbox will do at the start.
• Be sure to enforce the process. Ignoring the engagement point defeats the purpose.
• Get exec support and market your service. The engagement point, process, and key SME’s should be advertised like rock stars.
• Project support should be recognized in the list of services your team provides, whether or not you have a fancy service catalog.
• Metrics: how about

• % requests served within SLA
• # of requests +/- predicted per quarter e.g. we anticipated 30 but received 60!

• Get credit. Internal consulting should be included in your perf goals and review. cha-ching.

The ultimate goal is to serve the organization while enjoying your job.

I hope you find these useful. Please contact me or leave a comment with feedback or questions.

How Old Is That Vuln?

I root for vuln scanners to succeed (no pun intended).  Back in the day scanners helped automate a laborious task and they’ve continued to improve their products. However their fight in the marketplace is far from over. While vuln scanners own the vulnerability assessment market, for some reason they never quite finished delivering on vulnerability management i.e.  ensure unacceptable vulns are remediated per policy.

Today we’re seeing GRC and other vendors consume vuln scans like they’re commodity pork bellies.  They add on some workflow, connectors, reporting, and presto, you have the technology capable of supporting a great assessment service (almost as good as bacon).

This post has two purposes. One is to wax all nostalgic on how much I love vuln scanners. The other is to help explain why Third Defense just released a reporting extension on top of them. No, we are not providing all the workflow like the GRC crowd. We’re simply responding to a customer request who asked if we could help them out. Since I’ve been asking vuln scan vendors to produce this report for years, I said YES.

The report is simply to show vulnerability age compared to policy. It’s easy to age a vulnerability on specific hosts across scans. It’s also incredibly powerful for a security team to report on overdue vulns per asset group. The goal is to increase accountability and drive risk acceptance|mitigation decisions across business owners. Odds are you already have pre-negotiated mitigation time frames per sev level with the Ops group. Some folks include these time frames in policy. Others yet call them Service Level Agreements.

Either way, you now have an easy way to show vuln age. Here are the steps:

1. Paste your existing asset group names, owners, and IP ranges into the Vuln Tracker app
2. Assign a remediation date per group and sev level
3. Upload scans

The result is a simple histogram that can show:

• Age of all vulns across all groups
• All Overdue vulns
• Vuln age per group
• Overdue vulns per group

Here’s a screen snip (click to expand):

I fully expect vuln scan vendors to add in this report someday. It would be cool if we helped hasten the process. Until then, I encourage you to try out this report. The increased visibility into remediation performance works wonders. In my experience, maturing the vuln mngt process is the easiest across all of security. It’s rewarding to sit down with ops every month and translate vuln definitions into risk statements for your business. Take the next step and verify the appropriate remediation occurs.

As always, we welcome your feedback, even if it’s to remind us that we’re crazy to add a feature like this :)

Quick note: out of the gate we only support nessus. Let us know if you’d like to see more.

 

Will The Public Sector Show Us How It’s Done?

I’m still forming the thought here and want to get some feedback. I pose this question not because I think the public sector is better than private at managing risk. I pose it because it’s easier (sometimes mandated) that the public sector share data with the taxpayers. Which industry is going to step forward and start sharing examples of risk program structure, tools, deliverables, metrics, etc?

Thanks to Justin Somaini for his pointer to the University of CA ERM program. Yesterday I had Grace Cricket’s webinar on ERM running in the background. When something caught my ear, I’d check it out on UC’s fantastic Resources site. Below are some notes I jotted down. Huge disclaimer: I didn’t take the time to listen to everything or view all the resources. The purpose of the this post and these notes are to incite interest and Thank Grace and co for sharing!

Begin rough notes and soundbites:

“How do you know if you’re doing well?” Develop KPI’s, find the data. Instead of asking (stakeholders) what keeps them up at night. Ask them how they measure success. (me: Brilliant!). In the past, data was ad hoc and manual, not repeatable. Need technology to manage information. Selected IBM to develop solution.

- Developed ERM maturity model. Use S&P rating methodology.

- Retrospective reviews of impacts > $50k. (me: great source of evidence)

- ERM is basically a COE to cross pillars of risk management.

- developed an ERMIS: single portal, organizes information for monitoring performance.

- Developed Risk Assessment workbooks. (me: these are simple but a great starting point. I fully expect their use isn’t consistent, not always data driven, subject to politics, etc. but it’s a start for them)

- Probably could have spend a lot of money on fancy tools but folks are able to use simple tools, empower them to use something familiar e.g. xls

- combination of pull surveys vs. push assessments (me: the xls workbooks). complement by ERM maturity rating e.g. rims.org? 107 ERM activities across 5 categories. Self rate cmm like scale to rate ERM program itself.

- question from audience: what’s the motivation to start: 1. need an overall champion e.g. CRO. 2. need for a unified response to catastrophic incidents

- (52 mins in) ERM tracking: identified ~40 Ent. Risks across broad categories e.g. the one IT risk entry: Title “Decentralization of systems leading to data inconsistencies and fragmentation.
Mitigation: Senior leadership has recently put in place storage contols in this area;
Development and Maintenance Standards and (hard to hear?) local policies.
Data, Monitoring & Reporting: Reported at local level; Programing quality assurance and testing: approvals by programming managers and users before moving new systems or changes to production.

- advice: identify lowest hanging fruit, iterate

- about 700 KPI’s across ERM (me: I assume across the whole UC ecosystem)

- Risk Appetite: to begin, just show performance against average of institutions across UC system e.g. yellow is 5% of average.

- question from audience: Do you charge each institution for ERM services?  No, funded centrally.

- Can ERM quantify it’s benefit e.g. “elminimated cost of claims system @ $4M, ERM costs 2.5M/year. Improved credit rating.

- 1:08 in: First big win was workers comp. “cost of risk for fy09/10 reduced from $18.46 to $14.76″ ??

- first win was leveraging individual risk assessment tools. Next win will probably be to leverage the portal roll-up.

- How is CRO perceived e.g. auditors? Depends. Use qualitative to get past “sin factor.” Need to leverage qual and quant. Not an audit, no overlap from compliance. Focused on helping individual owners manage risk more effectively.

- CRO is a generalist to break down silos e.g. Financial, Safety, Compliance, IT Governance.

End rough notes and soundbites.

I don’t know if UC’s ERM program is good or bad. I do know I appreciate the insight. I also know we need more sharing. Please let me know as you come across great examples we can all learn from.

Thanks again Grace!

Source Seattle Slides & Goodness

I had a great time meeting new folks at Source Seattle. I must admit I hadn’t heard of Source before the Source Boston event. Many folks I follow on twitter (see @JaredPfost for a list) were all a-buzz about Source Boston. I had to reach out to Stacy Thayer (@StacyThayer) and join in the fun. I’m glad I did and look forward to helping Stacy and co. promote Source in the future. Boston may have success with professional sports but Seattle has better beer. If that’s not the foundation to develop the best security conference, I don’t know what is…

Anyway, here’s the official slideshare link to my session on How To Find The Right Amount of Security Spend.

Update: 30 min video here.

Here are the slides in .pdf:  How_Much_Security. Let me know if you’d like the source ppt slides (do you feel less comfortable downloading a .pdf or .ppt :).

Aside: one of my favorite talks was from Myles Conley http://www.sourceconference.com/seattle/speakers_2011.asp#mconley

Keep on the lookout for his slides when they’re available. Myles did an excellent job analyzing breach data to emphasize what roles infosec organizations really need i.e. web app dev’s may be cool but perhaps project managers may be more useful…

Metrics Rule!

I’ve written a lot about metrics, starting with their role and value in the Magnificent 7 series. I recently provided additional context in the Security Spending post. I’ll continue focusing on metrics until they become mainstream or I fall on my sword. I think they’re that important and if I took a CISO job tomorrow, evaluating the state of metrics would be one of my top three tasks. This post is inspired by email threads I had last week with a couple true security leaders who wanted to compare notes. I was also inspired by a great article from Pete Lindstrom on Building Out Your Strategic Security Metric Framework. It’s a great article but I want to make sure folks consider a couple additions.

The first is to recognize the resistance when building or expanding a metrics program. I haven’t heard anyone say they don’t want to measure. What I do hear is they don’t have enough time. Taking time away from business as usual and committed initiatives is tough so executive support is required. Do your executives really want to understand acceptable risk and efficiencies or not? If so, read more.

The second is the simple tactic of requiring each metric to have a target value. These targets define what “acceptable” means to asset and control owners. Targets are also a good forcing function to make sure a metric passes the “so what” test. Defining target values also normalizes % and integer based metrics. Stakeholders don’t have to understand every detail but they can quickly see if performance is above/below a committed level.

Now back to the inspiration for this post. Below is a list of metrics derived from a couple IT shops of my past and what I’ve seen/heard across my journey. You’ve seen me organize metrics by IT services and technical domains. This snapshot leverages the questions to define security investment: are we operating at acceptable risk, and are we as efficient a possible. I like this more business-friendly structure and for the true crazy elite, it aligns nicely into a balanced scorecard. Obviously I can’t include target values for you but please make sure you do.

Are we operating at acceptable risk?

Applications

• % internal projects using SDL
• % external projects with SDL in contract
• # pre-production (final security review) sev 1 security bugs
• # post-prod bugs for new/updated apps
• # security bugs identified in production application assessments
• % SDLC staff trained

Data

• % of PII encrypted/obfuscated per policy
• # of PII policy violations identified with DLP

IAM

• % time to complete User Access Reviews
• % of administrator accounts validated per policy cycle e.g. quarterly
• % user accounts managed through automation (or central directory)
• % of employees and non-employee workers with baseline access profiles
• % accounts deprovisioned per SLA

Vendor

• % critical vendors e.g. with PII, assessed per policy
• # of outstanding vendor risks

Device (aka scanners are your friend)

• % workstations/laptops outside patch SLA (e.g. critical 7 days, important 30 days, maintenance 90 days)
• % workstations/laptops with vulns outside SLA
• % workstations and laptops Managed for Security where “managed” is a bundle of healthy agents e.g. AV, HIDS.
• same as above for servers
• %  accuracy of cmdb vs network enumeration (scan)

Network

• # Network attached devices with vulns outside SLA
• % Accuracy of inventory (scan enumeration vs. cmdb)

Security Monitoring

• % network monitored per policy
• % servers monitored for security (may add a separate item for DB’s)
• % security tickets resolved within SLA

Incident

• # of critical incidents
• # of moderate incidents
• % incidents managed per SLA (identification, declaration, QFE, root cause, resolution)

People

• % staff trained per policy
• -Include outreach programs as appropriate e.g. cross-crash team meetings, brown bags, newsletters, sponsored socials, etc.
• Might be cool to break out social engineering incidents separately?
• # compliments showered on security team (wouldn’t that be nice)

Are we as efficient as possible?

Security Team Improvement

• % security tools and deployment rationalized (gauge of architecture discipline and utilization)
• # of security innovation ideas transformed into action (to encourage team participation) (yes, set a target value here too)

Security Program

• % Infosec spend aligned to strategy (which is/should be aligned to biz strategy)
• % security processes documented with assigned SLA’s (documented means RACI, swimlane, even SIPOC for all you nerds)
• % of key security processes with completed FMEA (failure mode exception analysis i.e. how can/do we break)
• % security initiatives completed on time
• % security initiatives completed on budget

Business Engagement

• % Line of Business security assessments completed on schedule e.g. semi-annual (aligned to budget cycle)
• % risks identified without status resolution (accepted, mitigated, mitigating). This focuses on risks identified during assessments and consulting e.g. evaluating cloud services, authentication requirements, etc.
• % consulting projects complete within SLA (volume and quality of LoB and IT projects supported with internal consulting)
• % capacity of internal consulting team (gauge supply/demand)

Compliance

• # “significant” audit findings identified
• % accepted remediation actions completed on time
• % audits following process (pre-scheduled, proper scope, duration)
• security violations managed per policy aka exception duration and risk acceptance by asset owner

…………..

Phew, looks like a lot of work and it is. If you really want to go crazy, create a performance roll-up for each section or the list as a whole. Target values normalize metrics so you can communicate % above/below expected progress. For perspective, it took us years working through the above and unfortunately I didn’t always get to see it through i.e. bank collapses :-) Plus, if your experience is like mine, your team will resist most of the above because they don’t have the time. It’s up to leadership to set the work priorities, reward improvement, dissolve heroism, and recognize the costs and benefits of metrics. In my experience, the start-up pain is well worth the reward. Not only will you have a solid foundation to measure, you’ll have a great source of evidence for your risk assessment process!

I encourage you to critique, add, subtract from the above. I’ll update the based on feedback and maybe crowd source this a bit. There’s no one right answer and metrics will evolve over time. Hope this was helpful!