How Decision-Makers Can Monitor Quality of Care
Approved by Council - June 1998
American Academy of Child and Adolescent Psychiatry
3615 Wisconsin Avenue, NW
Washington, D.C. 20016-3007
1998 by the American Academy of Child and Adolescent Psychiatry

Table of Contents

Authors and Acknowledgments
The American Academy of Child and Adolescent Psychiatry's Work Group on Community Systems of Care developed this document for Children and Adolescents with Serious Emotional Disturbances.

Work Group on Community Systems of Care:
Andres Pumariega, M.D., Chair
Nancy C. Winters, M.D., Outcomes Subcommittee Chair
John Diamond, M.D.
Emilio Dominguez, M.D.
Theodore Fallon, M.D., M.P.H.
Katherine Grimes, M.D.
Graeme Hanson, M.D.
William Heffron, M.D.
Robert Klaehn, M.D.
Ira Lourie, M.D.
Larry Marx, M.D.
Albert Solnit, M.D.
Deborah Thurber, M.D.

Work Group Consultants: Alan Axelson, M.D.
Mary Jane England, M.D.
Charles Huffine, M.D.

AACAP Staff Contact: Kristin Kroeger, Director of Clinical Affairs, 800-333-7636, ext. 108

The Academy thanks the Federation of Families for its thoughtful and helpful review.

Back to the Table of Contents

Preface
This document is intended for a broad audience of individuals and groups interested in organized systems of mental health services for children. While the document is written from the "system of care" perspective, it is not meant to diminish the fundamental role of the individual practitioner within the system of care. The system of care perspective represents a paradigm shift resulting from the recognition in the early 1980's that children with serious emotional disorders were receiving care in a fragmented treatment system. This recognition led to initiatives throughout the country to establish organized systems of care. A different vocabulary has evolved along with the systems of care, which broadens the traditional doctor-patient context by adding new perspectives, including those of patients and families as 'consumers' of care, and the child-serving agencies as 'stakeholders' in the system of care. In addition, the systems of care perspective emphasizes the values of 'family-centered,' 'community-based,' and 'culturally competent' care. One of the important contributions of the systems of care perspective is that it has allowed families to join with mental health providers as equal partners in creating innovative service systems that can be responsive to the needs of children, families, and communities.

Nancy C. Winters, M.D.
Chair, Outcomes Subcommittee
Work Group on Community Systems of Care
For Children and Adolescents with Serious
Emotional Disturbances

June, 1998

Back to the Table of Contents

Introduction
This document, a companion piece to the Academy's publication, Best Principles for Managed Care Medicaid RFPs: How Decision-Makers Can Select and Monitor High Quality Programs,1 is designed to assist state Medicaid agencies in evaluating whether managed Medicaid contractors are offering effective programs that adhere to the principles articulated in the agency's original request for proposal (RFP). This document also may be applied more broadly than to Medicaid vendors alone, since there is increasingly less distinction between public and private managed care. It is hoped that families, clinicians, and other community advocates for children may find this document useful when giving state Medicaid agencies feedback as to how well their managed care programs are performing.

Managed Medicaid mental health programs may have diverse goals relating to unique characteristics of their communities. These programs, however, should each strive to provide cost-effective, high-quality mental health services to children and families, to promote optimal health and development and to reduce the family and societal burden of serious emotional and behavioral disturbances. Unfortunately, current efforts to contain costs increase the risk of compromises in the quality of care. It is thus imperative that the purchasers, providers, and consumers of care take a leadership role in developing a process for holding managed care Medicaid programs accountable for the quality of the care that they provide.

There is a growing national consensus that the best indicators of quality of care are the results, or outcomes, of the interventions offered. A number of organizations have engaged in a national effort to develop appropriate outcomes indicators for managed care programs. While a complete listing of all the organizations involved in outcomes evaluation is beyond the scope of this document, among the most prominent are: National Committee for Quality Assurance (NCQA); Joint Commission on Health Care Accreditation of Hospitals (JCAHO); the Center for Mental Health Services' Mental Health Statistics Improvement Program (MHSIP) Consumer-Oriented Report Card; and practice guidelines and parameters by the American Psychiatric Association (APA) and the American Academy of Child and Adolescent Psychiatry (AACAP).

The various initiatives are limited, however, because the availability of scientifically tested methodology, especially for assessing outcomes in systems of care, has not kept pace with the demand for such outcomes indicators. Also, most of the outcomes initiatives were designed for assessing service delivery to adults, and therefore lack specificity for children and adolescents.

This document is not intended to duplicate or replace the many formats currently used for measuring outcomes and quality in managed care report cards or industry standards. Nor is it intended to supplant quality indicators or protocols used to measure outcomes in scientific research. Rather, the aim of this document is to provide pragmatic, broadly applicable information on useful indicators and potential instruments specifically focused on the unique needs of children, adolescents, and their families.

In the relatively new field of outcomes measurement, a number of terms are in use that are variously defined and sometimes overlap in meaning. Please refer to the glossary for discussion of how the terms 'principles,' 'outcomes,' 'outcome domains,' 'performance categories,' and 'quality indicators' are defined in this document.

1Best Principles for Managed Care Medicaid RFPs: How Decision-Makers Can Select and Monitor High Quality Programs (1996), American Academy of Child and Adolescent Psychiatry

Back to the Table of Contents

Guiding Principles
High-quality care has been defined as clinically appropriate, humane, and guided by the best interests of children and their families.2 The importance of assuring that high-quality mental health care is available to children and families cannot be overemphasized. It is known that childhood psychopathology can be associated with adverse long-term outcomes, including psychiatric morbidity, substance abuse, suicide, school dropout, poor vocational functioning, teen pregnancy, and incarceration.3,4 Such outcomes represent significant financial and emotional costs to families as well as to society as a whole. Early, clinically appropriate intervention may prevent, or at least mitigate, developmental damage from various risk factors, including psychiatric illness.5,6

The selection of quality indicators in this document was guided by principles of effective service delivery in systems of care, first articulated in 1986 in the Child and Adolescent Service System Program (CASSP) Guiding Principles for the System of Care.7 Interventions for children and families should be provided in an integrated, coordinated manner that is comprehensive and culturally competent. Care should be offered in the least restrictive environment that meets the child's needs, in a manner that supports the family and community's efforts to provide the best quality of life for children.

It is necessary to recognize that there are multiple stakeholders involved in a managed care Medicaid contract. These include the child and family, treatment providers, managed care Medicaid contractors, the federal government, state Medicaid agencies, and the many child-serving agencies in the community (i.e. public and private child welfare, education, juvenile justice, health, substance abuse, and mental retardation/developmental disabilities organizations); as well as members of the community in which the program resides. Each of these stakeholders may desire different outcomes. For example, the state Medicaid office may emphasize control of overall costs; members of the community may want increases to children's access to services; child welfare agencies may want to decrease the number of out-of-home placements; and families might want a reduction or resolution of their child's symptoms and an increased ability to function in various social roles. Each of these perspectives is valid and should be represented in the community's outcomes model.

In addition, a system of mental health care should take into consideration the special characteristics and needs of children and adolescents with severe and persistent disorders. Although these youth frequently are high utilizers of care, it is well documented that they tend to receive fragmented, poorly coordinated care in overly restrictive settings.8 A system of care that adequately addresses the needs of these youngsters should include integrated services, innovative financing, alternatives to hospital services based in the youth's community, and long-term intensive case management.9 One purpose of this document is to provide a basis for assuring that a high quality of care is provided for this very vulnerable population.

2Goldstein J, Solnit AJ, Goldstein S, Freud A (1996), The Best Interests of the Child: The Least Detrimental Alternative. New York: The Free Press.
3Offord DR and Bennet KJ (1994), Conduct disorder: long-term outcomes and intervention effectiveness. J Am Acad Child Adolesc Psychiatry, 33:1069-1078.
4Rutter M (1995), Relationships between mental disorders in childhood and adulthood. Acta Psychiatr Scand 91:73-85.
5Borduin CM, Mann BJ, Cone LT, Henggeler SW, Fucci BR, Blaske DM, Williams RA (1995), Multisystemic treatment of serious juvenile offenders: long-term prevention of criminality and violence. J Consult Clin Psychol, 63:569-578.
6Clarke GN, Hawkins W, Murphy M, Sheeber LB, Lewinsohn PM, Seeley JR (1995), Targeted prevention of unipolar depressive disorder in an at-risk sample of high school adolescents: a randomized trial of a group cognitive intervention. J Am Acad Child Adolesc Psychiatry, 34:312-321.
7Stroul BA and Friedman RM (1986), A System of Care for Severely Emotionally Disturbed Children and Youth. CASSP Technical Assistance Center, Washington, D.C.
8Knitzer J (1982), Unclaimed Children. Washington, D.C.: Children's Defense Fund.
9England MJ, Cole R (1992), Building systems of care for youth with serious mental illness. Hospital and Community Psychiatry, (43)6: 630-633.

Back to the Table of Contents

Elements of Successful Managed Care Medicaid Mental Health Programs
Successful managed Medicaid mental health programs provide services that are:

  • Child- and family-focused, with the needs of children and families dictating service mix and system design;
  • Embedded in a system of care within the child's community;
  • Culturally competent;
  • Responsive to the needs of high-risk children, including children with serious and persistent disorders;
  • Based on children's and families' strengths, as well as needs;
  • Coordinated with primary care;
  • Provided in the least restrictive setting that can meet the child's needs; and
  • Enforcing a no eject/no reject policy.

Successful outcomes measurement in these programs:

  • Is practical, as well as evidence-based, using measures with demonstrated validity and reliability;
  • Collects data regarding a child's functioning from multiple sources, including the child, family, school personnel, primary physician, psychiatrist, and other relevant agents;
  • Obtains comparable data for children not in the program, to allow for evaluation of program-specific results;
  • Includes patient and family (consumer) perspectives in the selected outcomes indicators; and
  • Acknowledges the objectives of different stakeholders in the system.
Back to the Table of Contents

Domains of Outcomes Measurement
Outcomes research is relatively new, and experimental designs with accepted scientific validity, such as randomized controlled trials, are difficult to apply longitudinally. In addition, results in clinical settings frequently have not matched results in laboratory or controlled research settings. Thus, it is not surprising that while research in systems of care thus far has demonstrated increased consumer satisfaction, at times it has failed to show superior clinical outcomes compared with traditional delivery systems.

There are a number of explanations for these results. The methodology for measuring clinical outcomes in populations receiving heterogeneous services, as opposed to individuals receiving a single, well-defined treatment, is not well developed. There may be a loss of fidelity to an established treatment when it is transferred to a 'community' sample. Also, change may occur over longer periods of time than the intervals currently measured. Systems of care need to operate for adequate periods of time to demonstrate favorable clinical outcomes. A growing body of literature, however, supports the clinical and cost-effectiveness of community-based systems of care (including those in Vermont and Ventura County, California) and of specific community-based interventions.

Measurement of outcomes requires a multidimensional approach in which a number of outcome domains are addressed. There is now reasonably good consensus about the domains that should be measured (see Table 1). It should be noted that family burden of illness15 includes economic factors. The cultural context of the child, family, and community also should be taken into account.16 Since children often are served by multiple agencies, integration and coordination of services are critical to achieving desired clinical outcomes, especially for children with serious and persistent disorders. For this reason, system organization as a programmatic outcome is included as an outcomes domain. The concept of stakeholders broadens the category of consumer satisfaction. Stakeholders in a system of care include patients, families, providers, and child-serving agencies, such as the education and child welfare systems.

Table 1. Outcomes Domains
Clinical Symptoms
Functional Status
Health Status
Quality of Life
Service Utilization
Family Burden of Illness
Stakeholder Satisfaction
Community Safety
Cost
System Organization

Since this document focuses on the performance of managed care Medicaid programs, the outcomes domains listed in Table 1 are addressed within broader performance categories that capture the quality and effectiveness of a managed care mental health system as a whole (see Table 2).

Table 2. Performance Categories
Access
Appropriateness
Effectiveness
Prevention
Cost

Access, appropriateness, effectiveness, and cost are performance categories commonly used in quality assurance (QA) plans. Prevention has been included as a performance category in this document because the prevention of later illness and functional impairment, and associated collateral costs, is particularly important in systems serving children and adolescents. Identification and early intervention are important secondary prevention activities, relevant both for individual children as well as for the program population. There is increasing recognition of the need for screening for mental health as well as general health disorders. Hypertension and depression are examples of disorders in which early identification can improve clinical outcome. Although primary prevention is not relevant for individual children already identified as seriously ill, as a programmatic population outcome, it is a potentially useful quality indicator. A successful program could lower the burden of illness in the community over time by better screening and allocation of funds.

10Weisz JR, Donenberg GR, Han SS, Weiss B (1995), Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology, 63 (5): 688-701.
11Bickman L (1996), A continuum of care: more is not always better. American Psychologist, 51: 689-701.
12Friedman R (1996), The Fort Bragg study: what can we conclude? Journal of Child and Family Studies, 5(2):161-168.
13Pumariega AJ, Glover S, (in press), New Developments in services delivery research for children, adolescents and their families. In: Olenick T, and Prinz R. (Eds.) Advances in Clinical Child Psychology.
14Hoagwood K, Jensen PS, Petti T, Burns B (1996), Outcomes of mental health care for children and adolescents: I. A comprehensive model. Journal of the American Academy of Child and Adolescent Psychiatry, 35:1055-1063.
15Burns BJ (1996), What drives outcomes for emotional and behavioral disorders in children and adolescents? New Directions for Mental Health Services, 71:89-102.
16Pumariega AJ (1996), Culturally competent evaluation in children's mental health. Journal of Child and Family Studies, 5(4):389-397.

Back to the Table of Contents

How to Use This Document
Quality indicators that may be associated with favorable outcomes for children and adolescents are outlined below, organized by five performance categories with outcomes domain subcategories. For each performance category, five indicators, marked with a diamond (˜), are considered appropriate starting points for quality and outcomes measurement. A suggested approach is to select two of the five starred indicators for QA, representing areas of performance expected to be in place early in the program. Three additional indicators then may be selected for ongoing continuous quality improvement (CQI). Over time, additional indicators can be added for CQI as measurement systems become more competent.

Back to the Table of Contents
Recommended Quality Indicators by Performance Category and Outcome Domain
Table 3
Recommended Quality Indicators
by Performance Category and Outcome Domain
* Five indicators in each Performance Category, marked with a diamond, are considered appropriate starting points for quality and outcomes measurements.

Performance Category
ACCESS
Domain Quality Indicator
Cost of Services
  • * Ratio of funds spent on children and adolescents out of total available funds for mental health, compared to ratio of pediatric population over total program membership
System Organization
  • * Access to a continuum of care (including chemical dependency, alternative intermediate-level services, intensive case management, etc.)
  • * Number of culturally competent clinicians and culturally appropriate/specific services (e.g., culturally normative parent guidance, use of ethnic community organizations)
  • Number of specialty children's services
  • Access to linguistic support services if needed or requested
  • Percentage of services provided in community settings (e.g., schools, home, community centers, churches)
Service Utilization
  • * Percentage of children seen by staff with specialized training in treating children, including the number of children seen by child and adolescent psychiatrists
  • Percentage of children served in other agencies (e.g., education, child welfare, juvenile justice) who gain access to mental health services, compared with baseline
Community Safety
  • * Time from referral to first appointment, both for routine and emergency referrals
  • Availability of child and adolescent psychiatrists after hours and for psychiatric emergencies
  • Availability of crisis intervention services, both general and specialized (e.g., for physical or sexual abuse; natural and man-made disasters, etc.)

 

Table 3 cont.
Recommended Quality Indicators
by Performance Category and Outcome Domain
* Five indicators in each Performance Category, marked with a diamond, are considered appropriate starting points for quality and outcomes measurements.

Performance Category
APPROPRIATENESS
Domain Quality Indicator
Stakeholder Satisfaction
  • * Satisfaction survey results for children, their families, the providers, and child-serving agencies. Suggested items for survey on parent/child involvement in developing plan of care: Were you present at the treatment-planning meeting? Did you agree with the plan? Do you feel that your concerns, opinions, etc., were addressed?
Service Utilization
  • * Percentage of funds spent on care in most restrictive vs. less restrictive settings
  • Percentage of patients seen in after-hours, crisis settings (e.g., emergency room)
  • Distribution of service types used compared with baseline
System Organization
  • Existence of an effective and efficient management information system (MIS)
  • * Percentage of children involved with multiple agencies who are served with pooled funds
  • * Adequacy of transitions between levels of service
  • Number of child-serving agencies that participate in multi-agency funding and planning
  • Evidence of the adequacy of the emergency response system, including after-hours procedures
Treatment Plan Appropriateness
  • Documentation of parent and child sign-off on treatment plans, including multi-agency treatment plans
  • * Documentation of link with primary care services
  • Documentation that treatment planning includes consultation with a child and adolescent psychiatrist
  • Documentation of informed consent for use of medications, with current medications (both psychotropic and non-psychotropic) indicated on the written treatment plan
  • Evidence that use of psychotropic medication meets generally accepted standards of care
  • Evidence that treatment plan identifies and addresses cultural and linguistic needs

 

Table 3 cont.
Recommended Quality Indicators
by Performance Category and Outcome Domain
* Five indicators in each Performance Category, marked with a diamond, are considered appropriate starting points for quality and outcomes measurements.

Performance Category
EFFECTIVENESS
Domain Quality Indicator
Functional Status
  • * Percentage of children per year placed outside the home due to emotional or behavioral problems
  • * Number of days per year spent in hospital or residential settings as a ratio over days in all services
  • * Number of arrests/legal referrals per year in population compared with community rates
  • * Number of children in restrictive school settings (including in-home tutoring) due to emotional or behavioral problems
  • * Improvement of impairment scores for a sample of patients in the program population (See Appendix B for examples of currently available instruments)
  • Number of days of school attended per year
  • Rates of graduation from middle and high school
  • Number of suicides and attempts per year compared with national and regional prevalence rates
  • Number of days in detention/incarceration per year
  • Number of child abuse reports and infanticides
  • Number of youth homicides
  • Number of teen pregnancies
Clinical Symptoms
  • Decrease in symptom level for a sample of patients in the population (See Appendix B for examples of available instruments)

 

Table 3 cont.
Recommended Quality Indicators
by Performance Category and Outcome Domain
* Five indicators in each Performance Category, marked with a diamond, are considered appropriate starting points for quality and outcomes measurements.

Performance Category
PREVENTION
Domain Quality Indicator
Family Burden of Illness
  • * Existence of screening mechanisms to identify family members at risk
  • Service utilization across different agencies by parents and siblings
Service Utilization
  • * Evidence of screening mechanisms to identify members of high-needs population who may be underutilizing services (e.g., persons with mental retardation or developmental disabilities)
  • * Number of preschool and early school-aged children receiving services as a trend compared with baseline
System Organization
  • Existence of ongoing health education efforts
  • Availability of advocacy or ombudsman function
Community Safety
  • * Evidence of screening mechanisms to identify high risk events/behaviors (e.g., suicide, violence, sexually transmitted diseases, substance abuse, teen pregnancy)
Cost of Services
  • * Funds spent on educational and preventive interventions (e.g., parent education, mediation, social skills training)
  • Percentage of funds available for mental health services per year spent on preschool and early school-aged children

 

Table 3 cont.
Recommended Quality Indicators
by Performance Category and Outcome Domain
* Five indicators in each Performance Category, marked with a diamond, are considered appropriate starting points for quality and outcomes measurements.

Performance Category
COST
Domain Quality Indicator
System Organization
  • * Percentage of overall costs for services vs. program overhead
Service Utilization
  • * Yearly costs of hospitalization and residential placement as a percentage of total mental health service costs child welfare, education, juvenile justice, chemical dependency, and other agencies)
  • * Total number of clients served vs. total funds spent (including funds spent by
  • * Number of new programs developed to fill services gaps
Cost of Services
  • Total cost of all services delivered to program population across all child-serving agencies (e.g., child welfare,education, juvenile justice, developmental disabilities)
  • * Costs of special education services due to emotional or behavioral problems
  • Costs of incarceration/detention/juvenile justice involvement
  • Costs of child welfare services for out-of-home placements
  • Measures of morbidity in the population referenced against costs

The list of indicators in Table 3 is not exhaustive, but reflects areas critical to serving children and adolescents. Some indicators may be more useful than others in particular communities. For example, some communities may lack child and adolescent psychiatrists or other specialized clinicians to serve the program population. Other communities may lack programs at less restrictive levels of care. The selection of indicators should be based on practical, as well as theoretical, considerations. Early in the development of an outcomes measurement program, indicators that are practical given community resources, and feasible, given available data management systems, should be emphasized.

Back to the Table of Contents

High Service Utilizers
Assessing the quality of services provided to children and adolescents who are high utilizers of services requires a specialized set of indicators. The following table of indicators, particularly relevant to children who are high-service utilizers, includes some items listed previously.

Table 4.
Indicators Particularly Relevant to Children
and Adolescents Who Are High-Service Utilizers
  • Availability of pooled funds, contributed to by all child-serving agencies;
  • Seamless transitions between levels of care;
  • Access to intermediate-level, alternative services (e.g., intensive home-based services)
  • Availability of a continuum of case management services (ranging from benefits management to intensive clinical case management);
  • Percentage of children requiring out-of-home placement (foster care, residential care, juvenile detention/incarceration) due to emotional or behavioral problems
Back to the Table of Contents

Additional Comments
Managed care Medicaid programs should establish and use a CQI process for the refinement of the service delivery system, based on ongoing outcomes data collection. Traditional QA methodology and data collection tools may be insufficient to capture the complexity of managed care Medicaid programs for children and adolescents with serious emotional disturbances. Programs may need to use flexible and innovative means to capture data. For example, using family advocates in the collection of consumer satisfaction data, establishing incentives to motivate stakeholder participation in data collection and analysis, and sampling a broad population base to determine system performance. In addition, managed care Medicaid programs should establish mechanisms for sharing information among child and family-serving institutions, while protecting confidentiality. Data sharing should be an ongoing procedure for CQI to support the larger objective of improved outcomes for the Medicaid population.

Empirical methods should be used as much as possible. Ideally, objective baseline data would be collected, and post-program data would be compared with baseline data and used in a CQI process to re-evaluate and modify the system. Although optimal intervals for outcomes measurement have not been empirically established, it is important that the indicators be applied over appropriate and adequate periods of time to allow for change, especially given that favorable clinical outcomes for more impaired children are not achieved rapidly. It also is important not to abandon current indicators when adding new indicators to the measurement system, so that longitudinal data can be collected.

Cost as an outcome should be viewed in the context of improving the functional status of the population. Children's services are more expensive than adult services because children require a multidisciplinary approach with multiple agency involvement and consultation. Children with severe and persistent disorders may not display significant symptom resolution, but rather a lack of deterioration. These children may need prolonged services, which also will affect cost indicators. In addition, cost indicators may initially increase in a managed care Medicaid program because more children receive services due to improved screening and access to care. Increased costs in mental health services should have a positive impact on health status, however, potentially providing a cost offset in primary care. Physical health costs should thus be reviewed in conjunction with mental health costs to capture any offset.

Back to the Table of Contents

Conclusion
The proliferation of managed care Medicaid mental health programs requires public accountability to ensure that high quality care is being provided along with cost containment. Although many national initiatives have addressed outcomes measurement, none has focused specifically on outcomes of mental health services for children and adolescents. This document identifies desirable outcomes and characteristics of service systems for this population and their families. Multiple outcomes domains that take into account the interests of different stakeholders are outlined. Specific, measurable quality indicators are provided, organized by the performance categories of access, appropriateness, effectiveness, and cost. Prevention as a performance category is added because of its particular relevance to children. Five quality indicators within each performance category are recommended for implementing a CQI process. Specific quality indicators for children with severe and persistent disorders (who also tend to be high-service utilizers) are provided. Empirical as well as pragmatic approaches to data collection are recommended and discussed. A glossary of terms, as well as resources and instruments for outcomes measurement, are provided in appendices A and B.

Back to the Table of Contents
Appendix A: Glossary
Outcome The product of an intervention. There are many different ways to look at the 'product' or outcome. There can be either clinical or functional outcomes. In a short-term, cognitive-behavioral treatment for depression, for example, the desired outcome is a significant reduction in depressive symptoms. However, if an adolescent with depression is less depressed after treatment (i.e. has a favorable clinical outcome), but is still not attending school, an important functional outcome has not been achieved. It can be argued that functional outcomes should carry as much, if not more weight than clinical outcomes.
There also are both individual and population-based outcomes. Examples of individual outcomes are those outlined above in the example of the depressed adolescent. Population-based outcomes might include community rates of incarceration, or community rates of school attendance. In order to measure population-based outcomes, a program must collect baseline data and then establish its own threshold based on that data. An advantage of using population-based indicators is that they may provide information about whether adequate numbers of children in the community needing services are being served. These different types of outcomes (e.g., clinical, functional) can be evaluated individually or together as subcategories or outcomes domains.
Performance Categories These represent a shift to a broader framework more common in industry than in medical literature. Whereas outcomes refers primarily to the end-product, which may take years to fully obtain, other areas of performance are recognized as important process measures in evaluating a system of care. An important reason for measuring areas of performance other than outcomes is that collecting outcomes data can be time-consuming and difficult. Also, there is far from perfect agreement as to what the most important outcomes are to measure. The end-product-only approach also does not provide a complete picture of how well the system of care is performing. For example, an essential feature of a system of care is that all children and families in need should be able to easily gain access to care. If outcomes only are examined, only those who have gained access to care would be represented in the data, thus leaving out a part of the covered population that is not receiving services. For these reasons, there is general consensus that aspects of the process of delivery of care should also be examined. Thus performance categories has been used to denote broad areas in which the system of care needs to be effective.
Principles The best interests of children and families guide fundamental aspects of a system of care that allow it to function in an ethical and humane manner that is guided by the best interests of children and families.
Quality Indicators The specific, quantifiable areas that are suggested for measurement. 'Quality' implies that appropriate care is delivered in the appropriate manner. Thus, quality indicators include aspects of the process. The term 'quality indicator' is used rather than 'outcomes indicator' because quality is a broader term, which denotes perceived value. Outcomes vary, especially for individuals, even when the best quality care is provided. Another reason for use of the term 'quality indicator' concerns scientific methodology: The data are incomplete concerning expected optimal outcomes for interventions believed to be of high quality. If complete data were available, it would be possible to look more at outcomes and less at the process to evaluate the quality of care.
Back to the Table of Contents
Appendix B: Examples of Available Instruments Useful For Outcomes Measurement
Functional Status
Child and Adolescent Functional Assessment Scale (CAFAS) (Hodges, 1990)
Children's Global Assessment Scale (CGAS) (Shaffer et al., 1983)
Residential Living Environment (Roles) and Placement Stability Scale (Pressley Ridge School, 1992)
Family Functioning
Family Adaptability and Cohesion Evaluation Scales (FACES III) (Olson et al., 1982)
Clinical Symptoms
Achenbach Questionnaires (Achenbach, 1991)
Child Behavior Checklist (CBCL)
Teacher Report Form (TRF)
Youth Self-Report (YSR)
Diagnosis (Structured Interviews)
Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS) (Puig-Antich & Chambers, 1978)
Diagnostic Interview Schedule for Children (DISC) (Costello et al., 1982; 1984)
Diagnostic Interview for Children and Adolescents (DICA) (Herjanic & Reich, 1982)
Symptom-Specific Scales
Center for Epidemiologic Studies-Depression Scale (CES-D) (Radloff, 1977)
Beck Depression Inventory (BDI) (Beck, 1961)
Conner's Parent & Teacher Rating Scales (Conners, 1969; 1973)
Yale-Brown Obsessive-Compulsive Scale - Children (Y-BOCS) (Goodman et al., 1989b, 1989c)
Revised Children's Manifest Anxiety Scale (FCMAS) (Reynolds & Paget, 1981; 1983)
Consumer Satisfaction
Client Satisfaction Questionnaire: CSQ-8 (Attkisson et al., 1989)
Family Empowerment Scale (MACRO)
Family Satisfaction and Needs Questionnaire (MACRO)
Back to the Table of Contents
Appendix C: System of Care Values and Principles Core Values for the System of Care
  1. The system of care should be child centered and family focused, with the needs of the child and family dictating the types and mix of services provided.
  2. The system of care should be community-based, with the locus of services as well as management and decision-making responsibility resting at the community level.
Guiding Principles for the System of Care
  1. Children with emotional disturbances should have access to a comprehensive array of services that address the child's physical, emotional, social, and educational needs.
  2. Children with emotional disturbances should receive individualized services in accordance with the unique needs and potential of each child and guided by an individualized service plan.
  3. Children with emotional disturbances should receive services within the least restrictive, most normative environment that is clinically appropriate.
  4. The families and surrogate families of children with emotional disturbances should be full participants in all aspects of the planning and delivery of services.
  5. Children with emotional disturbances should receive services that are integrated, with linkage between child-caring agencies and programs and mechanisms for planning, developing, and coordinating services.
  6. Children with emotional disturbances should be provided with case management or similar mechanisms to ensure that multiple services are delivered in a coordinated and therapeutic manner and that they can move through the system of services in accordance with their changing needs.
  7. Early identification and intervention for children with emotional problems should be promoted by the system of care in order to enhance the likelihood of positive outcomes.
  8. Children with emotional disturbances should be ensured smooth transitions to the adult service system as they reach maturity.
  9. The rights of children with emotional disturbances should be protected, and effective advocacy efforts for children and adolescents with emotional disturbances should be promoted.
  10. Children with emotional disturbances should receive culturally competent services that are sensitive and responsive to cultural differences and special needs and are provided without regard to race, religion, national origin, sex, physical disability, or other characteristics.

17Stroul, B. and Friedman, R. (1986). A System of Care for Severely Emotionally Disturbed Children & Youth. Washington, DC:CASSP Technical Assistance Center, Georgetown University Child Development Center.

Back to the Table of Contents

References
Achenbach TM (1991), Manual checklist/4-18 and 1991 profile. Burlington, VT: University of Vermont Department of Psychiatry.

American Academy of Child and Adolescent Psychiatry (1996), Best Principles for Managed Care Medicaid RFPs: How Decision-Makers Can Select and Monitor High Quality Programs.

Attkisson C, Larsen D, et al. (1989; 1990), The Client Satisfaction Questionnaire (CSQ), San Francisco: University of California.

Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J (1961), An inventory for measuring depression. Archives of General Psychiatry, 4:561-571.

Bickman L (1996), A continuum of care: More is not always better. American Psychologist, 51: 689-701.

Burns BJ (1996), What drives outcomes for emotional and behavioral disorders in children and adolescents? New Directions for Mental Health Services, 71:89-102.

Conners CK (1973), Rating scales for use in drug studies with children. Psychopharmacology Bulletin Special Issue, 24-84.

Costello AJ, Edelbrock C, Kalas R, Kessler MD, Karic SN (1982), The NIMH Diagnostic Interview Schedule for Children (DISC). Unpublished interview schedule. Department of Psychiatry, University of Pittsburgh.

England MJ, Cole R (1992), Building systems of care for youth with serious mental illness. Hospital and Community Psychiatry, (43)6: 630-633.

Friedman R (1996), The Fort Bragg study: What can we conclude? Journal of Child and Family Studies, 5(2): 161-168.

Goldstein J, Solnit AJ, Goldstein S, Freud A (1996), The Best Interests of the Child: The Least Detrimental Alternative. New York: The Free Press.

Goodman WK, Price LH, Rasmussen SA et al. (1989b), The Yale-Brown Obsessive-Compulsive Scale. I. Development, use, and reliability. Archives of General Psychiatry, 46:1006-1011.

Goodman WK, Price LH, Rasmussen SA et al. (1989c), The Yale-Brown Obsessive-Compulsive Scale. II. Validity. Archives of General Psychiatry, 46:1012-1016.

Herjanic B, Reich W (1982), Development of a structured psychiatric interview for children: agreement between child and parent on individual symptoms. Journal of Abnormal Child Psychology,10:307-324.

Hoagwood K, Jensen PS, Petti T, Burns B (1996), Outcomes of mental health care for children and adolescents: I. A comprehensive model. Journal of the American Academy of Child and Adolescent Psychiatry, 35: 1055-1063.

Hodges K (1990), Manual for the Child and Adolescent Functional Assessment Scale. Unpublished manuscript, Ypsilanti: Eastern Michigan University, Department of Psychology.

Knitzer J (1982), Unclaimed Children. Washington, D.C.: Children's Defense Fund.

Olson DH (1991), Commentary: three dimensional (3-D) circumplex model and revised scoring of FACES III. Family Process, 30:74-79.

Puig-Antich J, Chambers W (1978), The Schedule for Affective Disorders and Schizophrenia for school-aged children. Unpublished interview schedule, New York State Psychiatric Institute.

Pumariega AJ (1996), Culturally competent evaluation in children's mental health. Journal of Child and Family Studies, 5(4): 389-397.

Pumariega AJ, Glover S, New Developments in services delivery research for children, adolescents and their families. In: Olenick T, and Prinz R. (Eds.) Advances in Clinical Child Psychology, 20: In press.

Radloff LS (1977), The CES-D scale: a self-report depression scale for research in the general population. Applied Psychological Measures, 1:385-401.

Reynolds CR, Paget KD (1981), Factor analysis of the Revised Children's Manifest Anxiety Scale for whites, blacks, males, and females with a national normative sample. Journal of Clinical Consulting Psychology, 49:352-359.

Reynolds LR, Paget KD (1983), National normative and reliability data for the Revised Children's Manifest Anxiety Scale. School Psychological Review, 12:324-336.

Shaffer D, Gould MS, Brasic J, et al. (1983), A Children's global assessment scale (CGAS). Archives of General Psychiatry, 40:1228-1231.

Stroul BA and Freidman RM (1986), A System of Care for Severely Emotionally Disturbed Children and Youth. Child and Adolescent Service System Program (CASSP) Technical Assistance Center, Washington D.C.

Weisz JR, Donenberg GR, Han SS, Weiss B (1995). Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology, 63 (5): 688-701.

Back to the Table of Contents

Additional Resources on Outcomes and Performance Measurement
Evaluating the Outcome of Children's Mental Health Services: A Guide for the Use of Available Child and Family Outcome Measures. The Technical Assistance Center, Judge Baker Children's Center, 295 Longwood Avenue, Boston MA 02115.

Macro International, Inc. (Contracted evaluation agent for Center for Mental Health Services (CMHS) System of Care Projects). 3 Corporate Square, Atlanta Georgia 30329.

Final Report of the Mental Health Statistics Improvement Program (MHSIP) Task Force on a Consumer-Oriented Report Card. Center for Mental Health Services (CMHS) National Mental Health Services Knowledge Exchange Network, P.O. Box 42490, Washington, D.C. 20015. https://www.samhsa.gov/about-us/who-we-are/offices-centers/cmhs, 301.443-3343.

American College of Mental Health Administration (ACMHA), www.acmha.org, 412.244.0670.

American Managed Behavioral Healthcare Association (AMBHA), www.ambha.org, 202.434.4565.

Digital Equipment Corporation, www.digital.com, 508.493.9177.

Foundation on Accountability (FACCT), www.facct.org, 503.223.2228.

Institute for Behavioral Healthcare, www.ibh.com, 415.851.8411.

Joint Commission on the Accreditation of Hospitals (JCAHO), www.jcaho.org/perfmeas/about_pm.htm, 630.792.5889.

National Alliance on Mental Illness (NAMI), https://www.nami.org, 800.950.6264.

National Committee on Quality Assurance (NCQA), www.ncqa.org, 202.955.3500.

Substance Abuse and Mental Health Services Administration (SAMHSA), Office of Managed Care, www.samhsa.gov, 301.443.2817.

The Evaluation Center at HSRI, 2336 Massachusetts Avenue, Cambridge, MA 02140, woocher@hsri.org, 617.876.0476 x 326.

Back to the Table of Contents