Okay, so another uh technique that we can use in audits to look at evidence is something called technical verification. So this is where as an auditor you might want to actually validate the effectiveness of a technical process or control Because as much as you can read about technology in documents and you can talk about it in theory with people and so forth, sometimes you really want to see the technology Technology firsthand. So examples of technical verification might be analyzing configurations. So if somebody tells me, for example, that they filter certain traffic on their firewall at the perimeter of their network, well let's have a look at the config of that firewall and see what it's allowing in and out. If somebody tells me that they've implemented group policy in Microsoft to lock down the use of removable media then perhaps we can look at that group policy setting or even test with permission of course that that um access is uh uh being properly controlled. Things like viewing whether a backup can be restored, things like looking at vulnerability scans, etc. Now I must stress absolutely here, no matter what technical skills you may have as an auditor, auditors in an ISMS audit will never perform a hands-on test. So I remember an audit I went to at one point and I asked this person, there was an on-site server at the time, and I said, Can you tell me a little bit about um backups? Can you maybe show me the backup software and the backup schedule etc. And this person handed the mouse to me and said uh yeah you just you just look at what you want and even though I knew where to go and I knew how to use the uh software in question I actually said back to them I said look I said I can't do that I need you to show me or somebody else to show me and that's simply down to the fact that as an auditor if you do do anything hands-on yourself then there is a potential liability risk So the example I gave a moment ago there about um you know testing removable media, um what I'd be doing is looking at maybe the group policy and maybe asking an administrator to to uh to show me if they do have removal and show me the error message that props up when you plug it in or the fact that it doesn't work or whatever but that would be done by the auditee not by me But we can use technical verification to really get a good understanding and I I do like this as a particular piece of in evidence because for me you know we can talk all we want about security controls unless I kind of see them usually that's the real you might say proof in the pudding so to speak that what's being talked about, written about, is implemented technically speaking. Now there is a reference here to technical verification through the topic of scanning And depending on your background, you might be familiar with scanning, like vulnerability scanning. So some of you might have used vulnerability scanner tools like Nessus or OpenVAS, Qualis, these kinds of kind of things. Now usually in an audit we would not do this. So certainly in a penetration test or vulnerability assessment for sure. In an ISMS audit I certainly wouldn't be running scanning tools. The organization might, and they may be able to demonstrate to me the tools they use. Now there's also reference here to something called CATS or computer-assisted audit techniques So computer-assisted audit techniques are tools that can look for certain things. They can look for, let's say, user accounts that haven't been used. used for a period of time, accounts that have been locked or they can do data quality tests to see whether data in a database has missing information or duplicates etc. And I do see cats being used a lot in things like internal audit activities. Again, it's very, very rare that you as an external auditor would be used. using such a tool. So the message here is certainly we can ask the organization to show us the tools that they're using and even demonstrate them to us but we're not going to do that hands on on testing. The main thing for me is technical verification is about reviewing configs. Show me, demonstrate, let's see that evidence first hand. Now, let's imagine you've been out as an auditor and you've been gathering all this information, you've been interviewing people, you've been reviewing documents, you've looked at technical configs and done samples To get a reasonable level of assurance we have to undertake a technique regularly called corroboration. So you may have heard of this term before. And corroboration is about basically getting your information from multiple sources and making sure those pieces of information are set. essentially marry up. So we can do corroboration in a number of ways. We could ask several people the same type of question and see whether we're getting the same answers. Now the idea here isn't to, again, catch people out. It is about do those answers make sense? Are they consistent? We can of course compare interview responses and verbal information with documented information. So if you've explained a procedure to me, does that marry up with what we're saying in documents? We can compare the documented information or the notes we've been taking from interviews with direct observations. So we might have been told how a process works, but do we see something different when we see somebody applying it in in reality. And again, that might be a problem or not a problem that we're trying to corroborate. Or validating or maybe getting evidence from external sources So a good example of that might be on an audit recently the organization told me that they'd addressed a number of vulnerabilities on one of their pieces of infrastructure and they discovered these vulnerabilities through a pen test and so we looked at the pen test report produced by a third party This organization had done some remediation and there was another pen test report showing that that remediation had indeed been completed separately. So not only did I have the verbal assurance that this was going on. I also had then written assurance, but written assurance from a confirmative third party source. that corroboration give me a greater confidence that indeed those processes were happening, those issues were indeed being addressed. So that's what corroboration can do for us. Now we also need to evaluate the results of this evidence. What we're asking ourselves here is do we have enough evidence, enough evidence of a quality and a quantity to have what we call a reasonable assurance. So earlier in the course we talked about the idea of relevant information and the reliability, in other words, is the evidence speaking for itself? Who's collected the evidence? How recent is the evidence? what type of internal control systems the organization have. These all influence your thinking here. What you're trying to answer at this point as a lead auditor is based on the requirements of the standard, the audit criteria, do I have enough evidence that's relevant, reliable enough to make a judgment? Whether the judgment is that the organization is conformant or not, or something else, I need to be able to pull that together. What I've got to ask myself here in this phase is if I In my report, say to an auditee, this is my finding, can I, and anybody challenges me on that, can I then back that up? Can I say I'm basing that on this evidence? here and that's the biggest question. Unfortunately there is no scientific formula or mathematical number. A lot of this is down to your professional judgment and asking yourself Can I, for want of a better description, defend what I'm presenting? Now, one thing we said at the start of this section Is that we might be in a situation where we will audit what we call virtual activities or locations. So let me explain what we mean by that So, as we've seen the emergence of cloud technologies and the ability to work remotely, we're now finding, of course, more and more organizations don't necessarily have a lot of information or infrastructure physically on site. So I can think of a software development company I've audited recently. They're completely cloud-based, so there's no servers or infrastructure in an office somewhere that I can go and take a look at. at. And the individuals actually all work from home, so they don't necessarily have a physical building that they go to work to in every day. And in that case, we would consider their organization, at least part of it, to be what we call a virtual location. Now in ISO 27001 auditing when an organization applies to be certified, that's some of the information that the certification body will gather is details of the location, what's there, what's actually going on and so forth. So we are allowed in auditing where it's genuinely what we call a virtual location or remote site to conduct a virtual audit. And there are some guidance in ISO 19011 for conducting what we call virtual audits that we need to take into account. and things that we need to plan and agree in advance. So firstly if we're going to do an online virtual audit, the audit team should all be in agreement on the kind of technology we're going to use the kind of devices and software we're going to use. We'd want to of course agree that with the audit. So if we agree that we're going to use Microsoft Teams as our platform for talking to each other You know, let's agree on that. Let's make sure we're happy that we can use that particular platform. We'd need to agree in advance on things like how will you share documents with us and make sure that we're all using the same compatible software. We'd want to make sure in advance of such a virtual audit that the audit team and the auditee are doing some kind of technical tests to verify that things will work. I don't want to be turning up on the first day of the audit finding I can't connect or have a connectivity problem. We also need to think very carefully about contingency plan. I've been there before. What happens if you have a power cut or a network outage during the audit? You know, have you got a backup plan in this case? Do we need to put extra audit time in the agenda to deal with potential technical issues? These are all important factors that you wouldn't have necessarily encountered. in a face-to-face audit. Now ultimately, as I've said, we've got to think about those things on making sure both parties are comfortable. And ultimately the decision as to whether an audit will be done remotely, on site or a combination of the two will depend on circumstances. So like that example of organization I've told you about, because there is no physical location really other than a registered office, it makes perfect sense to do a virtual you load it. Whereas on with another organization I've worked with recently who are doing they don't necessarily have on-premise infrastructure but they're doing work at an office every day and they still have some paper-based information for them we will do some of the audit on site because we still need to go and look personally at the physical security environment. So it is a level of protection professional judgment do remember that we've got to get that balance right for if we're going to go online that we do do that testing make sure we're all comfortable with the approach in this case So in summary then what we've looked at in this section are the different procedures to collect evidence, interviews, document reviews. um sampling, observations, technical verification. These are all ways in order to get hold of and look at evidence. And then we talked about the fact that we would corroborate different pieces of evidence evidence and make a judgment as to whether we have enough relevant and reliable evidence in order to make a judgment on conformity. It is important that all those audit tests are thoroughly planned. Considered and later in the course we will look at audit test procedures. And of course, we need to think carefully now in the ever-changing world of cloud and remote working, how and when we audit remote sites. and ensuring that when we do that we are properly prepared so that we still have a positive audit experience and can still gather all the evidence that we may need in this