Outcomes Framework for Supervision: The Acid-Test

Since then, few researchers have examined the impact of supervision practices on client outcomes, with much of the supervision research focused primarily on the supervisor-trainee connection or attributes of the trainee (Bell et al., 2016; Morrison & Lent, 2018). Watkins’ (2020) subsequent review of the supervision research yielded similar results, and Tugendrajch et al. (2021) found the same dearth of studies examining this relationship in their review of the supervision research, suggesting that evidence supporting supervision’s impact on client outcomes continues to be insufficient and that there is still much to be learned about the connection between what supervisors do in supervision and therapeutic outcomes.

In a recursive progression, supervisors use the techniques that they learned from their supervisors, who in turn learned them from theirs. While practicing this time-honored guild succession model, following the cannons of their vocation, one must query whether their suggestions make people better. Can we have confidence that popular supervisory techniques such as isomorphism (parallel process) and the exploration of the person-of-the-therapist (Aponte & Ingram, 2018; Watkins, 2018; Callahan & Watkins, 2018) create a positive change for clients? In the same way, can one defend the focus on certain cherished components of supervision, such as self-awareness and reflectivity (Senediak, 2014), multicultural supervision (Gutierrez, 2018), and critical consciousness (Garcia et al., 2009)? Several studies have examined what trainees find helpful from their supervisors; however, there have been insufficient investigations into the link between what supervisors do and client outcomes (Hill et al., 2016; Tugendrajch et al., 2021; Watkins, 2020). The “yardstick” for patient outcomes is significantly biased in favor of subjective patient feedback measures relying on self-report data. McComb et al. (2019) warn that one must also be wary of abstracts and titles that claim to emphasize “empirically informed supervision.”

In a large-scale investigation, Rousmaniere et al. (2016) found that the supervisors in their study accounted for less than 1% of the variance in client outcomes, and a replication of this study found the same results (Whipple et al., 2020). Worse yet, the very clinical effectiveness of psychotherapy and psychopharmacologic treatments is being questioned; psychotherapy and pharmacotherapy yielded effect sizes in terms of standardized mean differences (SMDs) of about 0.30 or below in comparison with Treatment as Usual (TAU) or placebo, especially if effect sizes were adjusted for biases (Gomez et al., 2019; Cuijpers et al., 2020). Large effect sizes (⩾0.80) were only achieved in comparison of psychotherapy to weak comparators such as waiting list conditions (Feinstein et al., 2015; Cuijpers et al., 2020; Liu et al., 2017). These findings may suggest the field needs a more structured approach to supervision to ensure that the strategies applied by supervisors during supervision impact the success of treatment administered by their trainees.

Given the costs of mental health concerns to individuals, families, and society, it is imperative that we engage in supervision practices that directly attend to clinical outcomes.

This limitation of unvetted clinical supervision may well impair the very clinical effectiveness of psychotherapy, where the problems to be addressed are vast. According to a brief issued by the White House in 2022, between 2008 and 2019, the number of adolescents who reported at least one major depressive episode in the past year increased by approximately 90 percent. This is having a significant impact on the families with whom we work. Societal costs include increased homelessness and incarceration, worsening educational outcomes, decreased likelihood of entering the workforce, and increased domestic violence incidents, among others. Approximately $280 billion was spent on mental health services in the United States in 2020. Given the costs of mental health concerns to individuals, families, and society, it is imperative that we engage in supervision practices that directly attend to clinical outcomes.

Literature review

Although the focus of family therapy supervision is stated to be on an “outcomes” approach (Rigazio-DiGilio, 2014), the emphasis appears to remain on the knowledge and skills of the supervisee, as opposed to grounded data on how supervisees’ clients are progressing towards obtaining favorable outcomes. Sandra Rigazio-DiGilio’s (2014) contribution to The Wiley International Handbook of Clinical Supervision suggests that family therapy supervisors should focus on the conceptual, executive, and operational skills of their supervisees when evaluating their clinical work. Indeed, in-vogue models of family therapy supervision share a common focus on the supervisor-supervisee relationship and provide lengthy frameworks through which supervisors can develop a good understanding of their supervisees (Morgan & Sprenkle, 2007; Breunlin et al., 2011; Todd & Storm, 2014; Bernard & Goodyear, 2018; Robbins et al., 2018). While a focus on the therapist’s understanding and world view, and their functioning in a relationship with the supervisor, is undoubtedly essential, we hold that the field of family therapy would benefit from a shift in its supervision focus to client outcomes and the measures that indicate whether an intervention is working.

While it has been sparse, some researchers have examined the link between supervision practices and client outcomes with inconsistent findings (Keum & Wang, 2021; DePue et al., 2022; Rousmaniere et al., 2016), and supervision models have been proposed that focus on assessing client outcomes in various ways (Zhu & Luke, 2022; Watkins, 2018; Milne, 2014). The final step of Milne’s (2014) proposed model of evaluating supervision effectiveness is Systemic Outcome Evaluation, which calls for systemically assessing the client’s change by examining how the client’s change has practically impacted the system. This proposal suggests Results-Based Accountability (RBA), a model of supervision that provides a simple framework for doing just that.

Introducing Results-Based Accountability

Results-Based Accountability is a management framework that has transformed government services worldwide. Mark Friedman (2015), the author of RBA, having completed 19 years with Maryland’s Department of Human Resources as Chief Financial Officer, was dismayed at the vast expenditure of taxpayers’ money when there was a paucity of data on whether these programs were achieving required outcomes. In Friedman’s view, agencies typically used an “industrial model” like a factory. Resources were the input, and the only measurable output was the service delivery itself. In contrast, the RBA model is a change model, guided by questions like: Are we achieving the desired outcomes for our consumers? In the same way, we could ask our family therapy colleagues how supervision impacts client outcomes. RBA has been widely adopted and is utilized in all 50 states and many countries (M. Friedman, personal communication, 2023).

RBA’s framework is based on principles/tenets. To demonstrate the use of the RBA model, we introduce 16-year-old Mia.

 Tenet One—Bring all stakeholders together

The first step in RBA is bringing all stakeholders together. This is the duty of the therapist. In this case, the stakeholders are the family members, social worker/s, pediatrician (because of Mia’s anorexia nervosa), school nurse, state-appointed lawyer for Mia, the family therapist, and the family therapist’s supervisor. The mother and father, alleged perpetrators of Mia’s abuse, must be a part of the plan from the beginning. There was no imperative to emancipate this 16-year-old.

With young Mia teetering on fulminant anorexia and soon to be a runaway on the streets, the supervisor and therapist understood that the danger for this teenager was high. While originally part of a larger clinical team, the following will focus on the roles of the family therapist and family therapist supervisor.

 Tenet Two—Establish outcomes/objectives

Stakeholders determine the desired outcomes of treatment. Fragmentation of services can undo even the best of plans. We must ask, “In what way do we want this person/client to be better off? Moreover, how will we know in measurable terms the extent to which we achieve these desired conditions?” This perspective dramatically minimizes an unproductive, meandering adherence to clinical models and processes. After engaging with all stakeholders, the family therapist and supervisor established the desired outcomes. Mia’s weight and ameliorating her anorectic symptomatology were prioritized, as was eliminating family violence.

Tenet Three—Record data

Use data to improve performance. Friedman says, “Getting professional people to use data to improve performance requires a cultural shift” (Friedman, 2015, personal communication). Professional stakeholders with disparate backgrounds who may not be accustomed to data markers or the scientific orientation of tracking data can present a challenge. They sometimes struggle to understand that the most important measurements are grounded, objective, and observable.

The family therapist and supervisor agreed on weight as a measurement of the client’s progress toward reduction of anorectic symptomology. Mia’s biweekly weights were recorded and reported to the family therapist and pediatrician. The degree of triangulation in the family relationships and the number of incident-free home visits were the agreed-upon measures for reducing the propensity for family violence. The triangulation scale can be found in Table 1.

Tenet Four—IST/RBA Clinical Scorecard

The integration of Intensive Structural therapy (IST) (Fishman, 2013) and RBA principles helped create the IST-RBA Clinical Scorecard that measures performance over time. This valuable tool tracks the clinical effectiveness of the episodes of care and informs the family therapist and supervisor whether temporal goals are being achieved. The Clinical Scorecard should serve as a reveille, alerting the stakeholders when markers are askew.

The IST-RBA Clinical Scorecard empowers the family therapist and supervisor via the tracking of pivotal measurable parameters, increasing accountability for progress. The family therapist and supervisor collaboratively track progress toward established goals via this tool during each supervision meeting. It provides a mechanism for the family therapist and supervisor to focus directly on client outcomes. The Clinical Scorecard Worksheet with Mia’s information can be found in

Tenet Five—Plan of action 

A collaborative plan of action differs from the respective professionals’ individual plans. It is not an amalgamation of the plans but an agreed-upon single treatment agenda. The family therapist leads the transformation of the system. In the case of Mia’s anorexia, addressing the marital split and the ensuing triangulation (characteristics of psychosomatic families) will help to restructure the family toward a more functional system. The team notes that Mia’s increasing willingness to return home (number of incident-free home visits) indicated that the system was positively changing; the violence was diminished.

 Tenet Six—Use targets on scorecard

Targets serve as desired levels of future achievement, often alerting the treatment team to focus and enhance collaboration. They are specific, measurable, bound by realistic time periods, and based on previous performance. They are typically expressed as, “In x period of time, we want to achieve an x percentage of improvement in x behavior.”  Progress is measured by determining whether targets have been met, where targets measure realistic behavioral improvements. In Mia’s case, the family therapist and supervisor decided that her incident-free home visits in a one-month interval would show a 10% increase; in the second month, the targets may be adjusted. The targets indicate the interim outcome of the therapy. In the same way, weight gain against targets was also used.

Tenet Seven—The Homeostatic Maintainer

The Homeostatic Maintainer (Fishman, 2013) is an individual(s) or an interactional pattern that, when the system is perturbed, activates to prevent change, maintaining the status quo. When challenged by his wife, Mia’s father stormed out of the session, making it impossible to address and possibly resolve their marital issues at that moment. The goal of family therapy is to help the parents work together and cease triangulating their daughter. This dynamic is measured by the Triangulation Scale (TS). (Fishman, 2022). Triangulation is a central dynamic in anorexia and conduct disordered patients/clients. When using the TS, a low score indicates that the child’s behavior is more under control and the anorexia is more under control.

Tenet Eight—Turning the curve

This RBA tool tracks progress from the baseline and performance over time. It enables the family therapist and supervisor to adjust plans and interventions. A baseline is a starting point on a graph, representing historical data. It goes beyond the episodes of care to the complete course of treatment and informs the family therapist if they are headed in the right direction. Note that a common mistake, according to Friedman (2015), is to forecast where you want to go, not where you are going if you don’t do something more or different. Each client functions as their own control, and progress is defined as positively turning the curve from the baseline. An example of this can be found in Table 3, the Turning the Curve Tool.

Peter Drucker, the father of business journalism, famously wrote, “If you can’t measure it, you can’t manage it” (Zak, 2013, p.1). As therapists, we are managers of the system’s change; using grounded theory, we generate data. These measurements are utilized to track progress against a desired goal. In RBA, there is an implicit continuous cycle of review and refinement through this data-driven approach. Friedman (2015) says one of the crucial issues is picking the right performance measures. In Mia’s case, the family therapist and supervisor used the following measurements: weight gain, number of incident-free home visits as reported by Mia and the parents independently, and improved family functioning as evidenced by the triangulation scale. This approach provides a framework for linking supervision practices with desired client outcomes. 

Conclusion

We have introduced a simple framework for an organized system of care measuring and tracking outcomes regardless of the therapy modality. Transformation is focused on the contemporary social system. The IST-RBA Clinical Scorecard and the Turning the Curve tool, with fidelity to RBA principles, are the performance tools that inform the family therapist and supervisor of their effectiveness.

The Clinical Scorecard is utilized for each client and establishes objectives with a focus on targets and measurements. This provides observed grounded data for the family therapist and supervisor. The Turning the Curve tool provides a snapshot of progress for the family therapist and supervisor to review during each supervision meeting. This has not been done previously in supervision, as there were no suitable evaluation models for tracking outcomes (Watkins, 2020).

“The concept of competent practice and evidence-based practice are central considerations, and psychotherapy supervision now finds itself called to account,” according to Watkins (2011, p. 235). The reason for providing supervision and the ethical justification for requiring it is that it makes a difference with respect to client outcomes (Lichtenberg, 2007). We propose that this model, based on RBA and Intensive Structural Therapy can fulfill this need. Furthermore, using this process facilitates clinicians calling their supervisors to be accountable. Good client outcomes are the joint accountability of both parties, the family therapist and the supervisor.