4.2. Audit design matrices for selected water issues

When the audit objective is decided, it is time to start planning the audit. Deriving from audit focus, auditors have to phrase the audit objective (e.g. evaluate the effectiveness of protecting groundwater from diffuse pollution from agriculture).

Next, one needs to think about audit questions which need to be answered in order to meet the audit objective. Audit questions are usually broken into sub-questions. It is common to use closed (yes/no) questions in audit. Sub-questions are selected in such way that, if they are answered positively, the main question is also answered positively. 

If necessary, open questions (why?) are added to better understand the reasons for problems under scrutiny.

For evaluating the situation, each audit question is equipped with criteria - often deriving from legal acts, but also strategic documents and international agreements. Due to its rather detailed nature, the Water Framework Directive serves as a good source of criteria for the audit offices of the EU countries. 

Criteria describe the desired situation and are the main basis for the auditor’s evaluation. In case of absence of legal acts, auditors may use criteria which have been discussed and agreed upon with the auditee.

At this point, auditors need to consider which methods are most suitable for collecting and analysing information. Methods used in audits related to water management do not differ substantially from other performance audits. However, in case of a lack of data on water quality or the questionable quality of such data, external experts may be hired.

Audit design matrix (ADM) is a common tool widely used by auditors for organising audit questions, criteria and selecting suitable audit methods. Below you can see two audit matrices.

For more information on designing an ADM please visit MOOC: Introduction on Environmental Auditing in the Public Sector.

Table 4: Example of an audit matrix          

Audit title: Protection of groundwater from agriculture

Audit objective: Evaluate the effectiveness of protecting groundwater from diffuse pollution from agriculture

Audit question 1: Has the quality of groundwater improved over time?

Sub-questions

Criteria

Methods (incl. information sources)

Expected findings

1. Is the quality of groundwater monitored regularly?
  • National monitoring programme includes monitoring of groundwater in C % of wells at least in D years.

Review of monitoring data

Water quality is monitored only in Z region.

2. Is the quality of water maintained?
  • Water quality corresponds to the indicators of good water status

Analysis of monitoring data

Water quality does not meet quality standards in all wells. Deterioration of water quality has been noticed in Y% of wells.

Audit question 2: Has the government established requirements for preventing diffuse pollution?

1. Does the water law regulate diffuse pollution? 
  • The water law is harmonised to WFD;
  • water law includes provisions for preventing pollution from all households

Legal analysis

Water law regulates only pollution from large municipalities not from small households.

2.  Does the RBMP include targets for reducing diffuse pollution?  
  • Targets for reducing diffuse pollution as set in the RBMP

Legal analysis

Targets for major sources of diffuse pollution are outlined in the RBMP, but deadlines are very long (30 years).

3. Are there regulations for limiting the use of fertilisers and pesticides? 
  • Legislation includes provisions for the use of pesticides and fertilisers according to the WFD

Legal analysis

There is a good practice guideline for farmers.

Audit question 3: Are the regulations for protecting groundwater enforced?

1. Are the main polluters known to the government?
  • Pollution sources are analysed in an inventory to prepare the RBMP;
  • The environmental inspectorate is monitoring the pollution levels regularly and finds the pollution sources.

Review of the RBMP and its planning documents;

Interview with officials responsible for the RBMP;

interview in the environmental inspectorate

Some pollution sources that were recently discussed in the media were not known to the environmental inspectorate until its members got such information from newspaper sources.

2. Do farmers follow the good practice guideline?
  • 90% of farmers follow the good practice guideline

Online survey;

Interviews with 10 farmers (selected based on the online survey).

Some farmers follow the guideline; practice is varied.

 

Table 5: Example 2 of an audit matrix

Audit title: Drinking water management

Audit objective: Evaluate the effectiveness of the management of drinking water supply

Main audit question:  Is the provision of drinking water supply effectively organized?

Sub-questions

Criteria

Methods (incl. information sources)

Expected findings

1. Have legal provisions and a policy framework related to drinking water supply been established?
  • Systemic legal provisions and a policy framework are in place and address risks related to the supply and safety of drinking water;
  • Responsible agencies have been identified along with clear roles and responsibilities, and activities are coordinated

Analysis of relevant international and domestic legislation;

Document analysis (e.g. water management plans);

Interviews, questionnaires

A coherent legal framework is missing, and the activities of various agencies need to be more coordinated; Objectives set in legislation need clear expression through policy and planning instruments.

2. Is monitoring of drinking water supply effective?
  • Monitoring programmes have been established and are adequately funded;
  • Identified shortcomings are addressed and eliminated;
  • Quality standards for drinking water are met

Analysis of relevant legislation and documentation;

Interviews, questionnaires;

Data collection (e.g. monitoring data, permits)

Monitoring arrangements are in place, but there are shortcomings in the regularity and comprehensiveness of quality testing. Facilities for water quality testing need investments.

3. Is access to safe drinking water services ensured?
  • All public water supply systems meet quality standards;
  • Information on non-compliance and potential risks connected to the quality of drinking water is publicly available;
  • The price of drinking water is not an obstacle to drinking-water access

Review of monitoring data on drinking water supply systems;

Analysis of reasons for non-compliance (interviews with owners);

Review of publicly available data

Expert financial analysis of price of water

X share of the population is connected to water supply systems in which the drinking water does not meet quality standards; Corresponding data needs to be publicly accessible.

4. Are the pricing principles established in a financially sustainable way?
  • Service prices of water supply are set after evaluating required costs, and allow for necessary investments;
  • Continuity of the operation of water suppliers is ensured

 

Analysis of relevant legislation and documentation on pricing;

Interviews;

Data collection;

Review of international (best) practices

Tariffs must be revised as they will not cover the costs of water suppliers in the long term and rehabilitation of ageing supply systems may prove difficult without government subsidies. 

                                       

Table 6: Example 3 of an audit matrix

Audit title: Effectiveness of the financial programme X in protecting Y river basin

Audit objective: Evaluate whether the programme X for enhancing the status of water bodies in poor and bad status in Y river basin has been effective and efficient.

Audit question 1:  Has the status of water bodies, which were earlier in bad or poor status improved since the implementation of the programme X?

Sub-questions

Criteria

Methods (incl. information sources)

Expected findings

1.1   Was the programme designed according to the objectives of the Y river basin management plan?

  • Objectives of the programme X derive from the Y river basin management plan

Comparison of the objectives of the Y river basin management plan and programme X

Most objectives follow the Y river basin management plan.Some objectives of Y river basin management plan were omitted; however, they should have been included.

1.2   Has the chemical status of the water bodies improved?

  • water quality is regularly monitored
  • water quality has improved according to the standards established for chemical substances

Analysis of monitoring data

Interviews with experts

Water quality has somewhat improved. The level of some chemicals in some water bodies has remained above the level of standard

1.3   Has the ecological status of water bodies improved?

  • Ecological characteristics are regularly monitored

Ecological status has improved according to the standards established for biological elements and physicochemical characteristics

Analysis of monitoring data

Interviews with experts

Ecological status has somewhat improved in some water bodies. Endangered species have not returned to water bodies under study.

Audit question 2:  Was money used efficiently for managing programme X?

 

Sub-questions

Criteria

Methods (incl. information sources)

Expected findings

2.1   Were the projects selected according to the objectives of the programme?

  • Project selection criteria reflected the objectives of the programme
  • All project met the selection criteria

Review of project documentation (programme, project proposals, protocols of selection committee meetings)

 

Most projects follow the objectives of the programme, but some projects fail to meet the objectives.

2.2   Were the projects conducted according to conditions of the programme?

  • All project activities were carried out according to the plan
  • Money was used according to project plans

Analysis of relevant documentation (project reports, financial reports)

Interviews

 

Most projects were carried out according to the plan. Some activities were not carried out in some projects.

 

Tip for auditors

You may want to explore EUROSAI’s database of audits for summaries of water-related audits conducted by European Supreme Audit Offices: https://www.eurosai.org/en/databases/audits/index.html

  • Click See search filters in the middle of the page
  • Type „water“ in the open field under Search for text
  • Tick „Environment and natural resources“ under Filter by subject
  • Click Filter at the end of the search box

Global database of environmental audits maintained by the INTOSAI Working Group on Environmental Auditing may also prove an interesting and useful source: https://environmental-auditing.org/audit-database/