So in addition to conducting interviews and talking to people people. Another common evidence collection procedure of course is to review the documentation of the organisation. And I realise this has been discussed to some degree in the discussion about the stage one audit But just as sort of a of a recap here, we want to review all kinds of documentation from strategic documents like policies and procedures right down to actual records that might be there demonstrating that process is being followed and my main focus in here of course is the content that's the primary thing you know is the document what what is it telling me and does what the document tells me does that marry up for example to what I've been advised in some of these interviews. So let's say somebody verbally described to me the process let's say for background checking new employees, then I could re I could read the relevant procedure and say does it does this match up? Is there a you know mismatch or Is it consistent? I would still be looking at things like format and structure and also document management as well. You know, am I looking at the latest version? Does it look like this document's been subject? to any kind of review or quality control. So all the points that you covered in the discussion on the stage one audit would remain valid here as well. Now, as well as interviewing people and reviewing documents, one of the best in my opinion ways of really knowing what's going on in an organization is the concept of observation. So the audit procedure of observation is exactly that, as an auditor actually witnessing first hand for yourself without interfering in anything, various procedures. And that key term there without interfering is important. You know, we are not there as auditors to interrupt the business in any way. So what do I mean by observation? Well there are things we could ask to see. If there's software updates going on people could show me that happening If somebody is testing whether backup media has worked or not, they could potentially show me how they do the test. I could witness people logging into system. systems to see what access controls they've got or for physical audits, you know, going doing a physical on-site audit, looking at physical security you know, walking around the perimeter of the building, you know, experiencing the entry controls and seeing firsthand what's there are all types of observation. Now there are two types of observation just like there are two types of interview. We have the general observation and the detailed. So general of course is where you're looking at things. at a high level. So if for example I went to an office or a data centre and did a general tour to look at the facilities to understand what was there to observe some of the common physical security controls, you might say that's a general observation. If we are talking about a detailed observation, this is where we might want to see a specific task. In the example here it talks about the auditor witnessing the test of backup media. That's certainly a good detailed observation. An example there I can give, probably one of the most interesting ones. At one point I did an audit of a contact centre. So this contact centre had call centre operators taking telephone calls and dealing with customer inquiries, and that included processing credit card data. And so at the time, because they didn't have an automated system There was a requirement for the contact centre operator to stop the call and pause the recording when the credit card number was coming in. stop those numbers being recorded. As an auditor, what the organisation allowed me to do with authority, with non-disclosure agreements in place, etc. was basically sit with some of those contact centre employees and sort of witness how they did it and get them to just show me when they're handling a call the various screens they use how they log in etc And that gave me a really good appreciation and understanding of the technology and the processes and of course the security controls. that they had in place. All very good evidence to support what they'd told me verbally and what they'd stated in some of their documents. So when in order to carry out observations, when you see things, witness things, have things demonstrated to you, of course we've got to record these, especially if we are going to use those observations. as evidence. And the main way we do this is by taking notes. In fact, that's possibly in most cases the only way. So if I think to an audit I did recently, I did a physical security tour of the building and for each requirement of A7 in that case, I basically wrote the following. I said site two are conducted on X day at X time. The following controls on the perimeter of the building were observed such as, and then I talk about the vehicle barrier or the entry control or wherever it might be so written notes essentially photocopying documents might be one of those in reality uh I've never had the need to actually photocopy a document or take anything away. If there was a reason for doing so, then it is vitally important that you have the written permission and authority of the auditor. to do that. Similarly where it talks about taking photos, videos or audio. Now we've said quite rightly that that's not preferable. It's not preferable for several reasons. Some organizations have restrictions about media like that being on site, especially data centres and places like that, often you're not allowed to take recording equipment. in in the first place. Secondly, even if you are, there's the privacy question, you know, if you're shooting videos or taking pictures. If there's a real need to do so, then again, that's where we require permission. Now the only time I can think of really doing this in terms of taking photographs is when I've done audits that save networks and we've seen network comms cabinets that have been left open or you know badly cabled or something something like that. Sometimes it's easier to say there's a picture of what we've seen rather than sort of writing about it if you like. Sometimes the picture is more impactful, but again there has to be a reason for doing that and we have to have the authority to do so. So we can interview people, we can review documents, we can physically go and observe things or ask people to demonstrate things to us. Another thing we can do is analysis or sampling as it's sometimes called. And I just want to talk you through in this part a little bit about that procedure and things that you might want to understand. So there are controls and clauses within the standard where it's going to be very difficult to see everything everywhere. And therefore, you might, as an auditor, need to make a judgment On essentially a certain part of that process. So we say that analysis is studying a whole by studying its parts or taking a sample that's representative and then making a judgment. off the back of that sample. So we need to first to understand this talk about some principles and terminologies in the world of statistical analysis to be able to explain this. So we have four things, a population, a unit, a variable and a sample. So let's take an example. So the population basically is the set of people, objects, things that you want to see. The unit is each and every item of that population. The variable is the thing that we want to study, the thing that we want to look at. And the sample, of course, is the percentage or the group that we select in order to look for that variable. So we've got the example written in the slide here as an example. An auditor looks at 30 computers in a facility containing 500. So the 30 computers that auditor takes is a sample, the 500 is the population, and each computer is a unit. And in this case, the auditor is trying to validate that that computer has a proper A serial number and up-to-date anti-malware protection. So they're the two variables that that auditor's looking for. So in this high-level theoretical example, what the auditor is going to do when he or she's looked at those 30 computers. Is draw a conclusion. So even though they've not checked all 500, they've checked a certain amount. Now we'll debate about whether that sample size is a good or a bad one shortly, but all this is illustrating is that is what a sample. essentially is and there are controls in ISO 27001 that lend themselves very well to that kind of sampling. So things like competence and awareness In a large organisation, it's unlikely that I'm going to be able to interview all employees about their security awareness. I'm certainly going to have to take a sample. it's unlikely I'm going to be able to check all training records, so a sample would make sense. If we think about asset management in a large organization, let's say the the assets have to have some kind of characteristic like let's say laptops have to have whole disk encryption or you know have a serial number um attached to them like that example again I might take a sample Responding to security incidents. If the organizations have had several security incidents, I might want to analyse so many and so on. Protection against malware, I might not be able to check the malware signatures everywhere, but I might be able to take enough of a sample to draw a view, a conclusion if you like. Access control is another, let's say it's a requirement to be authorised to have access to a certain system. But you know you've got several thousand employees. I won't be able to look at every authorization, but maybe I can take so many to see and feel confident that those authorizations Are taking place. So there are places where we can do sampling. And what we intend to do here, based on the principles of ISO 17021, which are the guidelines for certification bodies. is take you through the things we need to think about when it comes to sampling in an ISO 27001 audit So as a recap, the reason we do this is because we cannot verify everything. And we're not looking to verify everything. What we want as auditors is a reasonable assurance that certain requirements are being made. met. So we're going to take a sample, but we have to look at sampling methods because so far that example you saw was very straightforward. It's a good one to explain what sampling is, but there are different types of sampling depending on statistics. statistical challenges and so forth. So before we talk about statistical sampling, there is another type of sampling that sometimes happens on audits which I have mixed feelings about but we do need to mention and this is called judgment-based sample So judgment-based sampling is where an auditor relies on their knowledge or skills or experience to choose a sample of things to look at. Now, I always say in an audit, especially management system audit, professional judgment is perfectly acceptable and is indeed a good thing You need to apply a level of professional judgment. The difficulty maybe with applying it in sampling is potentially it could lead to bias or confirmation of bias. So let me just give you a trivial example. Let's imagine I would like to look at the security of, let's say, user endpoint devices, maybe like smartphones and laptops and so forth. forth and according to your policy you know these devices should be locked down and controlled and so forth but based on my experience I know that sometimes, let's say, top management might not necessarily apply those controls, might be a bit more flexible in those controls. So I decide to take a sample of devices from the senior management population. Well, in one hand I'm using my professional judgment because I know that might be an area problem, but on the other, I could be accused potentially of not taking a representative sample of the whole organization, you know, maybe targeting a specific group or deliberately looking for non-conformity. So we've got to balance that. If you are going to use judgment-based sample do make sure as in insofar as possible that you've made a fair decision and you can justify that decision So the alternative to that is, as ITO 1911 points out, to look at what we call statistical sampling. And shortly we'll have a look at some of those statistical sampling methods and we'll look at the pros and cons of each. But before we do, there are things that can affect sampling that are important. Context size, nature and complexity of the organization, all of those that are valid. On certain audits, you know, if there's a complex environment, it might not be as simple as the example you just saw a moment ago. You know, taking 30 machines out of 500 and looking at them. Maybe all those machines have different variables. You know, maybe some of them are handling different types of data. So we may need to think about a sample that's representative of the landscape. The competency of the audit team makes a big difference. Do we have auditors who are capable of doing statistical-based sampling The time, you know, there is something we talk about in sampling, especially in internal audit, which is called the confidence level. And the confidence level grows to some degree the larger your sample size or the more areas it covers. But of course you have time limitations in an audit, so that can affect the type of sample you might be looking at looking at um and also how we deal with undesirable effects so there are some things to think about here Now another aspect of sampling before we go on to the different types in a management system audit is what we call multi-site sampling. So what do we mean by this? Well let's imagine the organization that's being certified or going through a certification audit has multiple locations. So I can think of one I work with right now. They've got Approximately 10 offices across the globe. Now when we do our initial audit, it's unlikely that we would visit all 10 locations. We may take a sample, we may look at certain geographical locations and that's what we call multi-site sampling. Now it's not necessarily about the amount of sites that we visit, but it's about the right mix to decide whether the management system is effective. So let me give you an example. This organization I'm speaking about has one large headquarters Another um location that does a lot of software development and then quite a lot of what we call small branch offices that are similar in style and nature in terms of what they deliver. So it makes sense that we will spend a lot of our time at the headquarters, at this development site, and then we'll take a sample of some of those branch offices. enough hopefully to have a reasonable assurance that the management system is working effectively. But we don't ignore the other branch offices. is in the audit following the surveillance audit then we look to visit those as well. But it's not necessarily about the number of places we've been, it's more about having that balance right