Planning and Reporting

Evaluation of the Women's Program

Volume I - Final Report

3.0 Methodology

This section of the report provides a detailed description of the methodology used to complete the evaluation.

3.1 Evaluation framework

The evaluation framework in Appendix C was prepared to guide the evaluation. The framework is organized around issues typically found in outcome evaluations: design and delivery, program impact, cost-effectiveness/alternatives, and relevance/rationale. The evaluation framework articulates the evaluation issues and questions and links these to the data collection methods.

3.2 Data collection methods

To enhance the reliability and validity of the findings, multiple data sources were used wherever possible to respond to the evaluation questions. Data collection methods included:

  • a review of program documents and administrative data
  • a review of program files (n=45)
  • a survey of applicants (n=215)
  • key informant interviews (n=41)
  • case studies (n=7).

Each of these methods is described in detail below. Throughout the report, the term "stakeholders" refers to a mixture of all groups from which information was collected, unless otherwise specified (e.g., external stakeholders refers to stakeholder groups outside the WP). The data collection instruments are in Appendix D. Detailed reports on the results for each method were prepared. These interim reports are provided in Volume II.

3.2.1 Document and data review

The review of program documentation and administrative data was intended to provide contextual information and to respond directly to many of the evaluation questions. The WP was responsible for providing relevant documentation and data to PRA. These materials include the Women's Program Procedures Manual, Accountability Framework (including logic model), Terms and Conditions, Risk-Based Audit Framework; program application and reporting forms; AGE and WP administrative data; Departmental Performance Reports, Reports on Plans and Priorities, consultation reports and prior evaluation and audit reports. In addition, PRA reviewed relevant documents from other sources within the Government of Canada, such as the Department of Finance, Treasury Board Secretariat, Department of Human Resources and Skills Development, Prime Minister's Office, House of Commons Standing Committee on the Status of Women, and the Policy Research Initiative (Privy Council Office), as well as information produced by the United Nations.

A report on the document and data review was submitted as a separate technical report, and elements of it have been integrated into this report. The AGE administrative data were summarized in a separate report on the AGE component of the WP, along with case studies of two AGE-funded initiatives (see Volume II).

3.2.2 File review

The purpose of the file review was, first, to identify the outcomes resulting from funded initiatives and to examine the extent to which these outcomes match with the expected results as first proposed by organizations; and second, to examine the extent to which the outcome-based approach to reporting, introduced in fiscal year 2003-2004, was being implemented and to identify any emerging issues that might benefit from further investigation or ongoing monitoring.

A total of 45 closed WP files from fiscal years 2001-2002, 2002-2003, and 2003-2004 were examined as part of the file review. This included 30 files from fiscal years 2001-2002 (n=17) and 2002-2003 (n=13), and 15 files from fiscal year 2003-2004. The 15 files for 2003-2004 represent all closed files for that year at the time the sample was selected.

For the purpose of the file review, the WP defined closed files as being either "closed complete" or "closed incomplete."15 For 2003-2004, all closed WP files meeting these definitions were included in the review. For 2001-2002 and 2002-2003, PRA randomly selected a sample of files (five from each of the six regions) based on annual lists of all funded files per region provided by the WP. Because these lists did not identify whether these files were closed, WP staff then reviewed the sample to identify any files that were not yet closed. Open files were replaced by PRA, and the process was repeated until all files in the sample were closed according to the definition above. This process, as well as the fact that the majority of the 15, 2003-2004 files were from the British Columbia/Yukon region, resulted in an uneven regional distribution of files, as shown in Table 2.

Table 2: Distribution of the files reviewed, by region
Region n=45 %
Atlantic 6 13%
Quebec/Nunavut 6 13%
Ontario 6 13%
Prairie/Northwest Territories 7 16%
British Columbia/Yukon 13 29%
National 7 16%
Total 45 100%

Note: An initiative is considered national in scope when it involves at least 3 of the 5 regions of SWC.

The uneven distribution across the country and limited number of files closed since the implementation of the outcome-based approach to reporting in 2003-2004 presents a significant limitation in terms of the file review's second goal, that is, to assess the extent to which the new approach has been implemented.

3.2.3 Survey of applicants

PRA conducted a survey of all women's groups and other equality-seeking organizations that applied to the WP in the last four fiscal years (2001-2002, 2002-2003, 2003-2004 and 2004-2005). The survey was intended to capture the opinions of applicants on the impact and current relevance of the WP and its components, as well as on program design and delivery and alternatives.

The survey was pre-tested in English and French with a small number of organizations before being mailed to all organizations composing the sample. The packages consisted of an introductory letter, explaining the purpose of the research and encouraging their participation; a questionnaire; and a self-addressed, postage-paid envelope. All organizations received both English and French packages. Respondents had the option of returning the survey to PRA either by mail, in the self-addressed, postage-paid envelope provided or by toll-free fax.

To develop the sample for the survey, PRA asked the WP to identify all organizations that had applied to the Program in the relevant fiscal years and for which the WP had contact information. The initial sample was compiled from various sources by WP staff and provided to PRA. However, when PRA reviewed the sample, numerous errors and inconsistencies were discovered, such as organizations appearing as both "funded" and "unfunded" applicants, duplicate entries, and similar problems. Once the problems with the sample had been resolved, PRA initially sent the survey to a total of 508 organizations, including 412 that had been funded at least once by the Program and 96 that had never been funded. The WP subsequently discovered that two funded organizations had been left off the original sample, and nine of the funded surveys and three of the unfunded surveys were returned to PRA due to incorrect addresses, reducing the final sample to 498.

Each survey had a unique identification number corresponding to a particular organization, enabling PRA to manage the survey process by conducting telephone follow-up with those organizations that had not returned their survey by a stipulated time. A total of 215 completed surveys were returned to PRA, for an overall response rate of 43%.

Table 3 provides a profile of survey respondents.

  • Two-thirds of respondents (64%) completed the survey in English, while the remainder (36%) completed the survey in French.

  • A large majority (89%) of respondents had received funding from the WP in the past. The remaining 11% had never been funded.

  • Respondents represented all six regions, with the largest proportions coming from the National and Quebec/Nunavut regions.

Table 3: Profile of survey respondents (n=215)
English 64%
French 36%
Funding status
Funded 89%
Unfunded 11%
National 27%
Quebec/Nunavut 22%
Prairie/Northwest Territories 16%
British Columbia/Yukon 14%
Ontario 10%
Atlantic 9%
Unknown* 1%

Source: Survey of women's groups and other equality-seeking organizations.
Note: Totals may not sum to 100% due to rounding.
*Two surveys were returned with the administrative code number obscured. As a result, the funding region could not be determined.

The information collected by the survey was data-entered and analysed using SPSS, a statistical software package commonly used in social science research. Results from the survey have been integrated into this report. The various sources and iterative approach used to compile the list of funded and unfunded organizations did not permit stratified sampling across key variables, hence it is not possible to assess whether the profile of survey respondents is closely liked to the distribution of the entire "population" of applicants to the WP.

3.2.4 Key informant interviews

PRA interviewed a total of 41 individuals with particular knowledge of the WP or general expertise in the area of women's equality. Similarly to the survey, key informant interviews were intended to capture the views of women's groups and other equality-seeking organizations regarding the WP, and included program staff and managers as well.

The WP was responsible for identifying appropriate key informants and providing PRA with up-to-date contact information. Key informants included WP managers and staff (n=15), program applicants (n=16), and external stakeholders (n=10). Throughout this report, the term "key informant" refers to a mixture of these three groups, unless otherwise specified.

PRA developed a separate interview guide for each of these three categories. The guides were tailored to the unique perspectives that the three groups of key informants were able to offer, but, wherever possible, similar questions were asked of all three groups. This approach helped the evaluation to establish how widely opinions are held across the various stakeholder groups.

Before being contacted by PRA, all key informants in the applicant and external stakeholder categories received a letter on PRA letterhead, describing the purpose and nature of the research and inviting their participation; program managers and staff were sent an email notification by the WP that they would be contacted. The letter stressed the importance of the study and the important role that each key informant played in its successful execution.

Approximately one week after circulating the introductory letter, PRA began contacting key informants by telephone to schedule interviews. Key informants had the option of completing the interview in English or French. A technical report summarizing the findings from the key informant interviews was submitted as a separate deliverable, and interview findings have been integrated into this report. Typical of key informant interviews, individual informants are able to speak about their own experiences and observations, but are rarely in a position to comment on a program's overall impact. It is not surprising that key informants had considerably more difficulty speaking to the WP's net contribution and its progress toward meeting its objectives, versus other topics discussed (more detail in Section 4.0 - Findings).

3.2.5 Case studies

PRA completed a total of seven case studies as part of the evaluation, including two case studies of AGE-funded organizations, as well as five case studies of organizations funded through "regular" WP funds. The case studies were intended to highlight the outcomes achieved by a sample of initiatives funded by the WP. The AGE case studies included one organization funded by the Ontario region and one funded by the National region (see report in Volume II). The remaining five case studies included one organization from each of the British Columbia/Yukon, Prairies/Northwest Territories, Quebec/Nunavut, Atlantic, and National regions. For each case study, the WP was responsible for securing the organization's agreement to participate in the research. PRA contacted the organizations selected for the case studies only once their agreement to participate had been secured.

Each case study consisted of a file review and interviews (completed by telephone) with one to three key informants. It should be noted that although the purpose of the case studies was to highlight outcomes achieved by selected initiatives, all of the case studies involved initiatives that were developed and carried out before the implementation of the outcome-based approach in 2003-2004. As a result, the documentation contained in the files did not clearly or consistently identify objectives or outcomes, or the extent to which outcomes (as opposed to activities) were achieved.

The results of the file review and the interview(s) were summarized into brief case study reports. The AGE case studies were submitted early in the evaluation as part of the report on the AGE component of the WP, while the remaining five case studies were submitted later in a separate report (see Volume II). Findings from all seven case studies have been included in this report.

15 Files defined as "closed complete" are files that were closed by the region/national office and include a regional/national assessment of the funded initiative. The assessment may be the Close-Out Assessment Form introduced in April 2003 or some other type of close-out assessment done by the region/national office before April 2003. Files defined as "closed incomplete" are files that were closed by the region/national office upon determination that the final report would not be submitted or is not adequate, and that further information could not be expected from the group.