Evaluation Process

Evaluation was a key component of TLC³ from the conception stage of the project and included the following elements:

• A cross-site process evaluation;
• A Management Information System; and
• Site-specific outcome evaluations.

The Foundation and the HDI team were interested in information on both process and outcomes - what programs the TLC³ sites delivered to children and parents, the organizational and community context in which they worked, and how successful the sites were in achieving overall project goals. Both the funder and the project managers also hoped to distill key learnings from the evaluation findings that could be of interest to, and shared with, a wider audience in the field of early childhood development.

In addition, the HDI team theorized that TLC³ could help local project participants gain a deeper understanding of the value of evaluation and acquire experience in doing an evaluation. To assist the sites with their evaluations and to pursue the TLC³ goal of creating bridges between research and the program delivery, the HDI team encouraged the sites to connect with a university-based researcher or evaluation consultant.

Cross-Site Process Evaluation

The HDI monitored the project sites to document that implementation was going as the sites had expected it would. TLC³ Director of Evaluation Nancy Cohen visited every site once a year, usually accompanied by the TLC³ Evaluation Coordinator. The site visits were intended not only to update the HDI team on how the sites were doing, but also to give the sites the opportunity to reflect on their experience in a systematic way. As the programs developed the focus of the site visits changed to reflect the current challenges and achievements.

The site visits provided a regular opportunity for Dr. Cohen to ask about challenges faced by the local program delivery teams. "Sometimes, just talking over things that may not be working as well as you had hoped provides that little extra nudge to make a change," she said. "First-hand observation also helps.

In addition to the site visits, the local projects were required to submit annual reports and annual budget statements to the HDI project management team in Toronto. The site leaders also presented oral and written overviews and updates, highlighting achievements and challenges, to their colleagues and others at the annual national symposia.

Management Information System

A Management Information System (MIS) was developed to gather descriptive information about participants in the various site programs in terms of demographics, child health and development, parent-child language and literacy activities. To gather this information, each participating family was interviewed by a member of the local TLC³ evaluation team when they began participating in TLC³. The MIS was not implemented until the 3rd year of the program so information was not gathered from all participants. Also at some sites parents chose not to complete the interview. The demographic information collected using the MIS interviews confirmed the diversity of the TLC³ participants at the different sites. Click here to see a brief profile of the participants at each site.

- Site Participant Demographics.

Site-Specific Outcome Evaluation

The original plan for the evaluation included a cross-site outcome evaluation that would use a common set of measures of language, cognition and school readiness at each site that could be compared to provincial and national statistics. The HDI team planned to design this evaluation with the advice of the national advisors, focusing on common elements across all the sites. The difficulty with this plan surfaced relatively early. The site programs were simply too different. They offered programs to different age groups, from infants to preschoolers. They operated in very different contexts. They delivered different programs. Nancy Cohen said "To impose a rigorous national evaluation protocol would have required us to strictly limit the program options of the local sites. That would have undermined our effort to support and strengthen the capacity of local families, organizations and communities to enhance children's early development."

Site-specific outcome evaluation was also part of the original evaluation plan, and those local evaluations proceeded. Each local site designed its own outcome evaluation and submitted its findings to the HDI team.

The project sites were assisted in getting started on their evaluation in the early phase of TLC³. At the annual symposium after the first year of implementation, they were given a logic model workbook that took them through a series of structured questions. It asked them to list what they expected to achieve for children, parents and (if relevant) other community members as a result of TLC³ programming; to articulate indicators or signposts for their anticipated outcomes, and the specific activities that were required to achieve their goals. Further steps included listing potential means of measuring outcomes and design issues.

The site teams worked on their logic model in their own small groups, with the support of a NAC member. The sites were not expected to finish the logic model on the spot, but Dr. Cohen wanted to ensure that they had received an explanation of the process and had made a good start on their model, with some expert help on hand to guide them.

Most of the site leaders were not researchers - they had experience in delivering services, but not in evaluation. They were expected to rely not only on advice from the HDI team, but also on their local research/evaluation advisors. Having to do their own evaluation put pressure on the local sites, but it also stimulated learning. They had some struggles with the process, but they emerged at the other end knowing a lot more about evaluation and its value. "It also helped several of the sites respond to government and other funders who wanted to know what they had achieved," Dr. Cohen said.