Outcomes in any outcomes hierarchy may or may not have one or more not-necessarily attributable (I[n-att]) or attributable indicators (I[att]. The comprehensiveness with which an indicator set "maps" onto an outcomes hierarchy has a number of implications for outcomes systems which are specified in the following indicator outcomes hierarchy coverage principles.
Principle: Indicators outcome mapping - Indicators (both I[n-att] and I[att]) should always be mapped onto their relevant outcomes hierarchy.
Discussion: The quality of an indicator set (either an I[n-att] or I[att] set) cannot be assessed until it has been mapped onto its underlying outcomes hierarchy. Such mapping has two purposes, firstly, it shows which areas of the outcomes hierarchy are illuminated by the indicator set. Secondly, it may provide guidance regarding how useful it is to collect particular indicators. Where indicator sets are presented without an analysis of their relationship to their underlying outcomes hierarchy there is no way of knowing whether or not they are a comprehensive indicator set.
Principle: Vertical indicator outcomes hierarchy reach - The higher up an outcomes hierarchy an indicator set reaches [both I[n-att] and I[att]) the better.
Discussion: If an indicator set (either an I[n-att] or I[att] set) only maps onto the lower reaches of an outcomes hierarchy it cannot provide routine information on whether or not higher-level outcomes are changing.
Principle: High vertical attributable indicators outcomes hierarchy reach - If attributable indicators (I[att]) reach up to high-level outcomes within an outcomes hierarchy there is general low, or no, need for whole-intervention high-level outcomes attribution evaluation (W).
Principle: Horizontal indicator outcomes hierarchy reach - The more comprehensively an indicator set reaches across an outcomes hierarchy the better (in terms of covering all high-level outcome areas).
Discussion: Often indicator sets (both I[n-att] and I[att] sets) will only reach across some of the high-level outcome areas within the relevant outcomes hierarchy. In such cases, the high-level outcome areas illuminated will obviously tend to only be those which are the most easily measured. Selecting suitable interventions just on the basis of their ability to change such a partial set of indicators is a strategic error. Assessing progress towards achieving the outcomes hierarchy as a whole on the basis of such a partial set of indicators results in the error of mistakenly focusing only on the easily measurable rather than on whether you are achieving the full set of priority outcomes [1].
Principle: High-level indicator priority where short outcomes hierarchy response time - Where an outcomes hierarchy has a short reponse time (i.e. high-level outcomes respond rapidly to an intervention) high-level outcome indicators within either set of indicators (I[n-att] and I[att]) are a higher priority for measurement than lower level outcome indicators.
Discussion: Collecting, analyzing and decision-makers focusing on indicators takes time. This principle shows which indicators are a priority for measuring. In a large under of short response outcomes hierarchies users often do not bother to collect any lower-level indicators, relying entirely on the highest-level outcome indicators.
Principle: Lower-level indicator priority where long outcomes hierarchy response time - Where an outcomes hierarchy has a long response time (i.e. high-level outcomes take a long time to respond to an intervention) lower-level indicator collection is a priority (first priority (I[att]) and second priority I[n-att]). [Provisional only].
Discussion: In the case of outcomes hierarchies with long response times, it is important to collect lower-level outcome indicators so as to have some information about what progress is being made. Because of the long outcomes hierarchy response time no timely information will be available from high-level outcome indicators.
Principle: Where whole-intervention high-level outcomes attribution evaluation designs
Notes:
[1] At this point some readers may ask: "well what are you expected to do apart from just measuring measurable indicators?". Indicators are routinely collected information about how outcomes are tracking. In outcomes theory they are the I[n-att] and I[att] building blocks of an outcomes system. In cases such as the one described under this principle, where there are important high-level outcome areas which do not have indicators, there are two alternative approaches which can be applied: the first is to undertake whole-intervention high-level outcome attribution evaluations (W building block) which may include measurement of those outcomes which do not have attributable indicators (I[att]); or, use additional lower-level evaluation (A) to build up a picture of what is happening in regard to causal chains leading up to the outcome areas which are not currently illuminated by indicators.
V1-0.