Findings from Four National Studies: Evaluating Systems Integration Approaches

Several national initiatives have been evaluated and provide useful information as to how frameworks were implemented and how effective the approaches were at improving services for children and families. The findings from those evaluations are summarized below.

Improving Child Welfare Outcomes Through Systems of Care

Overview of the National Cross-Site Evaluation (November 2010)

This 5-year initiative focused on infrastructure development to strengthen the capacity of human service agencies to support families involved in public child welfare. Concurrent with the initiative, the Technical Assistance and Evaluation Center (the Center) conducted a national cross-site evaluation of the demonstration program. The evaluation used a mixed methodological approach, which included a process and outcome component, to examine each grant community’s planning and implementation of its local Systems of Care initiative and the corresponding impact such work had on community collaboratives, agencies, and outcomes for children and families.

The cross-site evaluation found that grant communities progressed through initial infrastructure development and planning activities at varying rates, reflecting the presence or absence of a variety of start-up and readiness factors, including:

  • Clear understanding of the goals and focus of the Systems of Care initiative;
  • Experience with systems of care, cross-systems collaboration, and systems reform;
  • Strong and consistent leadership;
  • Shared vision for the community’s systems of care; and
  • While there was initial variation in readiness and capacity for implementation, the focus on building grant communities’ readiness through planning and TA activities appeared to enable less ready communities to build their capacity to undertake change.

Other lessons learned in the evaluation included:

  • Systems of care provide an overarching framework to coordinate and augment multiple systems and organizational change efforts within child welfare agencies;

  • An early focus on assessment and planning, coupled with intensive TA, can enhance communities’ readiness and capacity to implement systems of care;

  • Child welfare-led systems of care initiatives need strong and consistent leadership at the child welfare administrative level and project level;

  • As part of a comprehensive approach, systems of care require ongoing and multi-faceted implementation of six principles, including:

    • Interagency collaboration;
    • Individualized, strengths-based care;
    • Cultural and linguistic competence;
    • Child, youth, and family involvement;
    • Community-based approaches; and
    • Accountability.
  • Effecting systems and organizational change is a long-term process that requires ongoing commitment and investment.

Early Childhood Comprehensive Systems

Evaluation Results Brief (May 2011)

Since 2003, the Maternal and Child Health Bureau (MCHB) has funded the State Early Childhood Comprehensive Systems Initiative (ECCS), which supports State efforts to build comprehensive early childhood systems. ECCS required grantees to focus simultaneously on systems building across several different sectors, including early care and education, health care, mental health, family support, and parenting education.

ECCS’s evaluation highlighted the following key contributions in early childhood systems building:

  • State plans and frameworks spurred innovative efforts. As part of the initiative, States were asked to develop comprehensive systems plans covering the key components and systems elements. Most States reported that having a plan was a central factor in making progress in early childhood systems building. The plan served as a guiding framework for their efforts and helped their partners to think about ways to build connections across sectors and services. ECCS contributed to extensive progress on developing a system of social and emotional support for children and helped spur efforts in systems building in family support and parenting education.
  • Leverage and coordination of early childhood funding. One of ECCS’s most important roles has been to coordinate early childhood efforts in terms of planning and investment. This has led to greater coordination and more efficient use of resources at the State and community levels. ECCS helped coordinate federal investments in early childhood and make them more effective for resource planning by States and community agencies.
  • Ensuring health issues have a prominent place in early childhood discussions and planning. One of MCHB’s key reasons for funding ECCS was a sense that while health is a critical part of school readiness, it had not received the same level of attention as had cognitive development, primarily because the discussions centered on the early care and education sector. ECCS played an important role in increasing the prominence of health issues, including physical health, social and emotional health, and, in some States, oral health. The evaluation study demonstrated that early childhood partners are now more likely to raise issues related to the health of young children when discussing school readiness and child development.

Early Education Partnerships: State Actions and Local Lessons

Press release on study from the QUILT project (March 2003)

The Education Development Center, Inc. (EDC) released a study that explored different ways States and local early education programs used funding and resources to improve the quality and access of child care services for eligible families. The evaluation was based on an analysis of State and local partnership data compiled by the EDC-based Quality in Linking Together: Early Education Partnerships (QUILT) project. Results of the study found that ongoing partnership efforts can be facilitated by both States and local child care providers.

States can provide incentives—such as program grants, increased reimbursement rates for providers, and direct funding to providers—for those providers involved in partnerships. States can also take action to improve coordination among State agencies to reduce program barriers that deter partnership growth in the child care and early education sector.

Local providers can also support partnerships by promoting professional development opportunities for teaching staff, offering greater parenting support to families, developing agreements to better outline partnership roles and responsibilities, and providing financial strategies to anticipate those issues that arise when programs blend funds and resources.

The study concluded that by States supporting partnerships among various local child care programs, these coordination efforts can improve the quality and services for both children and families while providing a more effective use of funding for child care and early education services.

The First Seven Years: The BUILD Initiative and Early Childhood Systems Development

Historical Overview of the BUILD Initiative in the States

The BUILD Initiative, a project of the Early Childhood Funders’ Collaborative, helped States create comprehensive early childhood systems, developed community level systems building and public and private partnerships, and addressed equity and diversity issues and data and information systems. BUILD facilitated States to continue strategic thinking and action in specific areas, such as developing quality rating and improvement systems, supporting family, friend, and neighbor care, and early childhood health and mental health services.

Evaluation of the BUILD Initiative showed evidence that:

  • Greater success and sustainability of State efforts can be attributed to emphasis upon a long-term, organic approach that builds individual leadership as well as systems capacity, the use of interactive evaluation strategies, and collaborative approaches with other initiatives and organizations, including networking through a learning community that enables individual State leaders to provide TA to their peers.
  • In terms of investments needed to succeed with future early childhood systems building challenges and opportunities, the highest returns may result from those investments that foster State leadership through learning networks and TA that can inform and strengthen national systems.
  • The BUILD Initiative viewed continuous improvement and learning from QRIS as essential to establishing a quality early childhood system. The evaluation overview also validated the importance of facilitation and TA providers in the development of a comprehensive early childhood systems framework.