MOOC: Auditing Water Management

4.2. Audit design matrices for selected water issues

When the audit objective is decided, it is time start planning the audit. Deriving from audit focus, auditors have to phrase the audit objective (e.g. evaluate the effectiveness of protecting groundwater from diffuse pollution from agriculture).

Next, one needs to think about audit questions which need to be answered in order to meet the audit objective. Audit questions are usually broken into sub-questions. It is common to use closed (yes/no) questions in audit. Sub-questions are selected in such way that, if they are answered positively, the main question is also answered positively.

If necessary, open questions (why?) are added to better understand the reasons for problems under scrutiny.

For evaluating the situation, each audit question is equipped with criteria – often deriving from legal acts, but also strategic documents and international agreements. Due to its rather detailed nature, the Water Framework Directive serves as a good source of criteria for the audit offices of the EU countries.

Criteria describe the desired situation and are the main basis for the auditor’s evaluation. In case of absence of legal acts, auditors may use criteria which have been discussed and agreed upon with the auditee.

At this point, auditors need to consider which methods are most suitable for collecting and analysing information. Methods used in audits related to water management do not differ substantially from other performance audits. However, in case of a lack of data on water quality or the questionable quality of such data, external experts may be hired.

Audit design matrix (ADM) is a common tool widely used by auditors for organising audit questions, criteria and selecting suitable audit methods. Below you can see two audit matrices.

For more information on designing an ADM please visit MOOC: Introduction on Environmental Auditing.

Table 4: Example of an audit matrix

Audit title: Protection of groundwater from agriculture
Audit objective: Evaluate the effectiveness of protecting groundwater from diffuse pollution from agriculture
Audit question 1: Has the quality of groundwater improved over time?
*Sub-questions*	*Criteria*	*Methods (incl. information sources)*	*Expected findings*
1. Is the quality of groundwater monitored regularly?	National monitoring programme includes monitoring of groundwater in C % of wells at least in D years.	Review of monitoring data	Water quality is monitored only in Z region.
2. Is the quality of water maintained?	Water quality corresponds to the indicators of good water status	Analysis of monitoring data	Water quality does not meet quality standards in all wells. Deterioration of water quality has been noticed in Y% of wells.
Audit question 2: Has the government established requirements for preventing diffuse pollution?
1. Does the water law regulate diffuse pollution?	The water law is harmonised to WFD; water law includes provisions for preventing pollution from all households	Legal analysis	Water law regulates only pollution from large municipalities not from small households.
2. Does the RBMP include targets for reducing diffuse pollution?	Targets for reducing diffuse pollution as set in the RBMP	Legal analysis	Targets for major sources of diffuse pollution are outlined in the RBMP, but deadlines are very long (30 years).
3. Are there regulations for limiting the use of fertilisers and pesticides?	Legislation includes provisions for the use of pesticides and fertilisers according to the WFD	Legal analysis	There is a good practice guideline for farmers.
Audit question 3: Are the regulations for protecting groundwater enforced?
1. Are the main polluters known to the government?	Pollution sources are analysed in an inventory to prepare the RBMP; The environmental inspectorate is monitoring the pollution levels regularly and finds the pollution sources.	Review of the RBMP and its planning documents; Interview with officials responsible for the RBMP; interview in the environmental inspectorate	Some pollution sources that were recently discussed in the media were not known to the environmental inspectorate until its members got such information from newspaper sources.
2. Do farmers follow the good practice guideline?	90% of farmers follow the good practice guideline	Online survey; Interviews with 10 farmers (selected based on the online survey).	Some farmers follow the guideline; practice is varied.

Table 5: Example 2 of an audit matrix

Audit title: Drinking water management
Audit objective: Evaluate the effectiveness of the management of drinking water supply
Main audit question: Is the provision of drinking water supply effectively organized?
*Sub-questions*	*Criteria*	*Methods (incl. information sources)*	*Expected findings*
1. Have legal provisions and a policy framework related to drinking water supply been established?	Systemic legal provisions and a policy framework are in place and address risks related to the supply and safety of drinking water; Responsible agencies have been identified along with clear roles and responsibilities, and activities are coordinated	Analysis of relevant international and domestic legislation; Document analysis (e.g. water management plans); Interviews, questionnaires	A coherent legal framework is missing, and the activities of various agencies need to be more coordinated; Objectives set in legislation need clear expression through policy and planning instruments.
2. Is monitoring of drinking water supply effective?	Monitoring programmes have been established and are adequately funded; Identified shortcomings are addressed and eliminated; Quality standards for drinking water are met	Analysis of relevant legislation and documentation; Interviews, questionnaires; Data collection (e.g. monitoring data, permits)	Monitoring arrangements are in place, but there are shortcomings in the regularity and comprehensiveness of quality testing. Facilities for water quality testing need investments.
3. Is access to safe drinking water services ensured?	All public water supply systems meet quality standards; Information on non-compliance and potential risks connected to the quality of drinking water is publicly available; The price of drinking water is not an obstacle to drinking-water access	Review of monitoring data on drinking water supply systems; Analysis of reasons for non-compliance (interviews with owners); Review of publicly available data Expert financial analysis of price of water	X share of the population is connected to water supply systems in which the drinking water does not meet quality standards; Corresponding data needs to be publicly accessible.
4. Are the pricing principles established in a financially sustainable way?	Service prices of water supply are set after evaluating required costs, and allow for necessary investments; Continuity of the operation of water suppliers is ensured	Analysis of relevant legislation and documentation on pricing; Interviews; Data collection; Review of international (best) practices	Tariffs must be revised as they will not cover the costs of water suppliers in the long term and rehabilitation of ageing supply systems may prove difficult without government subsidies.

Table 6: Example 3 of an audit matrix

Audit title: Effectiveness of the financial programme X in protecting Y river basin
Audit objective: Evaluate whether the programme X for enhancing the status of water bodies in poor and bad status in Y river basin has been effective and efficient.
Audit question 1: Has the status of water bodies, which were earlier in bad or poor status improved since the implementation of the programme X?
Sub-questions	Criteria	Methods (incl. information sources)	Expected findings
1.1 Was the programme designed according to the objectives of the Y river basin management plan?	Objectives of the programme X derive from the Y river basin management plan	Comparison of the objectives of the Y river basin management plan and programme X	Most objectives follow the Y river basin management plan.Some objectives of Y river basin management plan were omitted; however, they should have been included.
1.2 Has the chemical status of the water bodies improved?	water quality is regularly monitored water quality has improved according to the standards established for chemical substances	Analysis of monitoring data Interviews with experts	Water quality has somewhat improved. The level of some chemicals in some water bodies has remained above the level of standard
1.3 Has the ecological status of water bodies improved?	Ecological characteristics are regularly monitored Ecological status has improved according to the standards established for biological elements and physicochemical characteristics	Analysis of monitoring data Interviews with experts	Ecological status has somewhat improved in some water bodies. Endangered species have not returned to water bodies under study.
Audit question 2: Was money used efficiently for managing programme X?
Sub-questions	Criteria	Methods (incl. information sources)	Expected findings
2.1 Were the projects selected according to the objectives of the programme?	Project selection criteria reflected the objectives of the programme All project met the selection criteria	Review of project documentation (programme, project proposals, protocols of selection committee meetings)	Most projects follow the objectives of the programme, but some projects fail to meet the objectives.
2.2 Were the projects conducted according to conditions of the programme?	All project activities were carried out according to the plan Money was used according to project plans	Analysis of relevant documentation (project reports, financial reports) Interviews	Most projects were carried out according to the plan. Some activities were not carried out in some projects.