So we need to make sure though as auditors if we're going to do sampling and we're going to base an audit finding off the back of a sample that we can demonstrate that we've done what might be called adequate sampling Sadly, however, ISO 17021, ISO 19011 does not say adequate sampling looks like this. Here's the number. Now if you work in other audit fields like financial auditing, for example, often there are precise formulas that you're required to follow. Things like stop-go sampling, things like um confidence and coefficient levels and so forth. I'm not going to go into these here, not because we don't want to talk about them, but they're less relevant in a management system audit context. But if you're doing wider information system auditing as part of a financial audit, this may be something you want to explore. What I'm going to present here is the the different types of sampling method and the sampling process. So we're going to sort of walk you through this. So we've got seven steps in terms of defining what qualifies as an error right through to documenting the sample process. So we're going to step you through these seven pieces step by step to explain the key points here. So the first thing we want to do when we think about it sampling is Define what we might consider an error. In other words, something that might affect the audit objectives. That's your first step. So if we see this example here, so it says the following. An error in sampling is an ISMS audit could occur if the auditor selects a sample of employees for verifying conformity to clause 7. 3 on awareness, but the sample is not representative of the workforce. So in other words, if I pick somebody out who's not actually doing the same job as somebody else, for example, you know that could be potentially an error or I pick somebody out who's not an actual employee but maybe who's a freelancer or something like that we might get the wrong picture. So we would define what could potentially be an error in this case before we start. We also want to think about the potential risks. What's the chance of perhaps not seeing the full picture, picking on the wrong areas and so forth? Now this is important and a big potential risk is in areas where you don't understand the landscape very well. So let's take a hypothetical scenario Let's imagine I want to look at the security arrangements, maybe let's say authentication, for an for applications. And I simply pick on a number of software applications that your organization uses. Now if I don't understand the fact that some of them have differences, I could run into a problem But what I mean by that is let's imagine there are some software applications that deal with highly sensitive information and some that deal with commercial information. They may have different security settings because of the kind of data they handle. If I don't know the difference between those and I just pick random uh applications, I might get the wrong idea. I might end up picking purely those that have handle sensitive information and not seeing that the others don't have the same controls in place. So I need to consider these potential risks before we start Another thing we need to be clear, and this leads into a similar thing, is to define the sample population. So we know what a population is in general, but over what period of time? So for example, let's say that I want to audit whether or not new users being added to a system are properly authorized Should be clear about which system I'm talking about, how many users are we talking about, over what period of time do I want to look at this evidence so we can clearly say in writing this is what the population looks like like. All changes to the ISMS performed in the last six months are part of the population. We're very clear about where that population starts and stops. But we need to understand, as per my point a moment ago, the characteristics of that population. So let's go back to my example of checking whether people have been authorized to have access to a particular system. Now it's not quite as simple as saying, well, there's a thousand people who've been granted access over the last year so I'll just have a look at so many because maybe some people have been granted administrator access, maybe some people are super users and maybe some some people are standard users Obviously, I'm keeping this example very simple, but the point is they then would all have different characteristics. So I probably then need to change my sample to fit that. So let's just say Keep it a bit simpler and this is very simplistic but but it hopefully gets the point across. Let's imagine there are a hundred users 10 of them are administrators, 10 of them are super users, and the other 80 are normal users. Okay, well if I'm going to choose a sample then based on those characteristics I want my sample to be representative of that. In other words, 80% of my sample should be focused on normal user accounts 10% of my sample should be focused on administrator accounts and 10% of my sample focused on superuser accounts. In other words, if I understand the characteristics of the the population I can have a sample that represents that population because if I don't I could get a very skewed view of reality So with that in mind, we have a little workflow here talking about selecting the sampling method. And depending on your view, you can follow this thro So we get this first question, do we intend to extrapolate the results? In other words, learn more from the results that we get And then of course we can be asked what type of extrapolation we can do. So when we say yes, often this works in the field of monetary audits or financial audits In our case, many of our approaches will be looking for specific cases, you know, looking for specific. security controls in place. So mainly in this selection we're often going for what's called monetary unit sampling Which is not necessarily about money, just to be clear, but what we call random sampling. So I think the best way to explain this is to have an exploration of the sampling methods and their their benefits, their purposes. So what I'll do here is I'm just going to run you through a few different sampling methods with an example of each and you can think about how you might apply this in an ISMS audit. So let's take the first, random. Okay, so very simple to explain, maybe harder to do in reality, but let's go back to my example of access control. And let's imagine we have a hundred users who have been granted access to an application. And I simply say, let's run a random selection. Let's just say I want to look at 10 of those just for the sake of argument. And I just say right I'll have any 10 from that pool of 100 using some kind of random number generator Or something like that. So total selection based on chance. I'm not interfering with uh which uh user forms I take a look at. So statistically quite reliable, but It may not be representative of the population if I don't think about some of the other characteristics, but it's the easiest to understand I could consider a systematic sample. So let's say there's a 100 users and I want to look at say 10 of them and I say okay give me A user from the first block of 10, a user from the second block, you know, 11 to 20, a user from the third block. So in other words, I'm selecting certain samples at certain intervals It is systematic because we're following a pre-described system to maybe try and get examples from across the sample There's still some randomness in that because I'm not saying give me number two, give me number eight, give me number twelve, etc. But I'm saying within this range I'll take some n However, if we have the situation like the one I explained earlier where of these hundred users, you know, ten of them are administrators, ten of them are super users, and the rest normal users, for example. and I want to select a sample that is representative of the population, that's where we call this stratified selection So that's where I want to make sure each subset is represented. So I want to see the administrator group, I want to see the superuser group and the normal use normal group. It could lead to false conclusions if I get my numbers wrong. So if I don't know for example that 10% of those users are administrators I could overly select users from the administrator group, for example, and get the wrong view of reality. So there is a risk of false conclusions unless you are fully familiar with the population and select a population that's representative. Another thing you could consider in a case like this is what's called block selection. Now this really isn't ideal because it's not really representative. This could be in an environment where you really don't understand it to some degree the population. So let's say, let's take a different example. Let's imagine I walk into a physical data center and there's various network communication capabilities. containing cabling and I want to look at cabling security and not knowing the differences between these I just say I'm going to go and look at that group over there And I'm going to base whatever I see based on that group. I've selected a block. That, you know, in in an environment where it might be difficult to understand what's happening, that can be useful but it may not necessarily be representative. So I think these are some of the the key points to consider. So to be clear these are the different types There's no right or wrong. ISO does not say you must use any of these, but you must be able to justify that when you put your ideas forward. So what we haven't talked about though is determining the sample size. This is something on a course like this I'm always being asked, well how many items should I take? Now unfortunately I can't give you a gold number. I can't say well every time you go in do 10% or 20% etc etc. It will vary. It does depend on the peculiarities of the ISMS audit and things like the audit objectives, the size of the population, which of those sampling methods you're going to choose to use maturity of processes, all of these things and the most important one arguably, the willing or the level of risk the audit team is willing to to accept. What do I mean by that? The level of risk related to errors or detection risk. Remember detection risk is the risk of not being able to detect something. So the smaller the sample size, usually the greater the chance of of course not picking up on anomalies or issues. So this will also depend on what we agreed up front with the audit in terms of materiality Now, obviously it wouldn't be very satisfactory of me to say to you, well, ISO 17021 and ISO 19011 have nothing to say about sample size, so let's move on. I don't think many People would be satisfied with that. So, to answer that question, what we have done is referred you here to a sampling table that was designed by the IT Governance Institute So the IT Governance Institute have given some quite good guidance on this. And what they're saying is depending on what you're auditing and the frequency According to their statistical analysis, these are the minimum keyword their minimum number of selections we should consider. So if I look at the top one where I'm seeing, let's say I'm going to audit some change management. and let's just say there's a change being made every day for the last year. I'm going to look at least at 25. Now that's the IT governance Institute's minimum recommendation, we may use more if we need to have a greater confidence. And I'll come back to that in just a moment. So let's imagine you're the auditor, you have selected a sample, you've chosen your method, you've chosen the size, you then evaluate the results Now, that doesn't mean we decide simply by looking at the results that something's conformant or not. We would record the results and make a decision On what to do with them. Now, usually in a classroom-based course, I'd usually fire a question at the audience and the question goes along this line. What I tell people is: imagine I've done The audit, let's say the uh example at the beginning of this section. And I've looked at those 30 computers out of the thousand, I think it was that they mentioned or 500. And let's imagine that 29 out of 30 all have up-to-date anti-malware signatures and a proper label on them, but one of them doesn't. I always ask the class What would you do? Would you consider that to be 29 out of 30 is good? We're conformant, that's a good result. Would you say, well, you found one that isn't? that's a non-conformity or would you take a different route? And what I tell people is two things. Number one, you can't just say it's conformant or it's not based on that. You've got to look at the context You know, what does the implications of one machine, let's say, missing their anti-malware signature mean in reality? How serious would that be when I look at the bigger picture And what I could do as well as answering that question, if you've got time, you could potentially expand your sample a little bit. So if I expand it a bit, is it still one? That's not got an up-to-date signature, which might imply an anomaly. Or actually am I discovering more? And statistic statistically, if I'm discovering more, that would point to a wider problem. So this is why we don't make a snap judgment. Based on a sample, but we do try and evaluate those results. And it is important as an auditor in your audit working papers to document that process, laying out Um in the audit report what sampling method you used, what the population was, what the sample size was uh in terms of the statistical percentage. percentage. Any nuances you took into account, so the reasoning and any any um if you like if you've applied a stratified method, what the stratified groups were, for example. what confidence you have in this overall. Now this is why I would say we'd never just rely on a sample anyway. We combine sample results with other evidence. But it's important to point this out because ultimately when we're deciding what to say to an auditee, you know, you're conformant, you're not conformant. etc then I've got to be able to explain my logic so we do have just to summarize this a good example of a systematic Sampling exercise using that table that you saw earlier from the IT Governance Institute. And here's the example that we're talking about. Let's imagine I'm an auditor and I'm wanting to look at so many change requests. To see if they've been authorised properly, been through the proper process of review and risk assessment. And there's been 400 of those in the year. So according to the IT governance table I should be looking at least 25. I might decide to look at more if I want a bigger confidence level but we'll start with 25. So I take the 400 divided by 25 that gives me this number 60 So what we're saying here is I'm going to choose one item every 16th step. But I'm not just going to go number 16, number 32, number 48, etc. I might start with a random number, so the fifth. 16 items on would be 21, another 16 items on would be 37 and so forth. So in other words, this is a systematic sample uh example if you look back at that table. Now this is only an example. This is not us advocating that that's how you do every sample, but hopefully it gives a good worked example here. So hopefully that's given us a good overview of what sampling is about. I realise statistics it can be a complex topic. There could be more to it than this, but this should give you a very good starting point. to be able to select samples in an ISMS audit that you have confidence in.