Evidence-Based Innovation: Criteria for Evaluating Lay Legal Assistance Programs

Elizabeth Chambliss

It is an exciting time for access to justice and access to justice research. Jurisdictions around the country are experimenting with new models for expanding access to legal assistance by training nonlawyer advocates and advisors to provide limited legal services in areas of high unmet need. Such models range from for-profit programs for specially trained paralegals to not-for-profit community-based programs using a variety of staffing models.¹ Research on such programs is growing and becoming more organized, rigorous, and impactful.² Finally, after decades of resistance to new categories of providers, regulators are beginning to pay attention to evidence about the limits of the lawyer-only model and possible benefits of expansion.³

The question now is, how are these programs working? And what should be the criteria for assessment? How might we move beyond case-by-case, after-the-fact program assessment based on the number of intakes and outcomes and incidents of demonstrable harm, toward a more robust, comparative framework for research? Two new articles tackle these questions by proposing specific evaluative criteria, measurement strategies, and sources of data to guide researchers and policymakers in program evaluation and design.

In Measures of Justice: Researching and Evaluating Lay Legal Assistance Programs, Tanina Rostain and James Teufel draw on assessment methods in healthcare and social services to define and compare three criteria for measuring the quality of legal services: “fidelity,” which measures providers’ adherence to preestablished work standards and procedures (P. 1491); “harm avoidance,” which focuses on consumer protection from specific harms in commercial contexts (for instance, bad results, unintentional default, and unnecessary purchases) (P. 1493); and “effectiveness,” which compares the services of different providers based on the user’s goals and experiences. (P. 1495.)

The promotion of a user-centered approach to assessment is a defining feature of both papers and marks an important shift from focusing only on providers’ perspectives. “Effectiveness” from a user perspective is measured not only by the legal outcome but also by the visibility and accessibility of the service; the timing of service (for instance, prevention versus crisis intervention); user trust in and the approachability of providers; and how users are treated by formal institutions such as agencies and courts (i.e. procedural justice). From a user perspective, Rostain and Teufel argue that the co-location of legal and adjacent services may be especially effective for increasing visibility, trust in providers, and “upstream” intervention, as illustrated by the widespread success of medical-legal partnerships. (P. 1498.)

Rostain and Teufel also define and compare various measures of program impact, with a focus on measures of interest to funders, such as financial and social return on investment, which measure monetary costs and benefits, and “cost effectiveness,” which includes measures of nonmonetary benefits such as well-being, employability, and perceptions of fairness. (P. 1504.) They conclude by calling on providers and regulators to define evaluative goals and measures as a component of program design:

Rather than an exercise after-the-fact, evaluation needs to be built into the design of innovations to capture the goals new models are intended to further…. The time for the American legal profession to develop an evidence base for its institutions and practices is long overdue. (P. 1507.)

In addition to designing for program “effectiveness,” Matthew Burnett and Rebecca Sandefur urge providers and regulators to design for “scalability” and “sustainability.” (P. 1529.) Based on a review of contexts in which lay legal assistance has long been allowed—immigration matters, Tribal courts, and jailhouse lawyering—as well as a variety of recently-approved community justice worker programs, A People-Centered Approach to Designing and Evaluating Community Justice Worker Programs in the United States defines effectiveness, scalability, and sustainability as the three core criteria for evaluating such programs; explains how each can be measured; and provides a detailed appendix linking research questions to specific sources of data. Burnett and Sandefur illustrate their framework by comparing programs on a series of measures related to the criteria they propose.

The focus on scalability is a key contribution of the paper.⁴ Regulatory barriers to nonlawyer assistance have led to mostly hyper-local, hyper-regulated pilot programs with significant barriers to entry and expansion. Only recently has the demonstrable effectiveness of trained lay providers in various contexts begun to penetrate the bar’s default resistance to new categories of providers, and many regulators remain skeptical. But while against this backdrop any move to expand effective assistance is exciting, locally handcrafted programs can be hard to scale. Thus, even as regulatory approval remains an important hurdle, Burnett and Sandefur urge providers and regulators to think beyond the threshold question of effectiveness and design programs for scale. As they argue:

Effective services alone cannot address the enormous justice gap unless they can scale to serve as many people as possible over the long term. Scalability involves increasing the impact of the activity while maintaining or increasing its effectiveness…. (P. 1530.)

In assessing programs’ potential for scaling, Burnett and Sandefur identify four barriers that must be overcome: barriers to entry, barriers to replication, barriers to learning, and barriers to funding. Barriers to entry are “what is required to engage in the authorized activity.” (P. 1531.) For instance, programs that require lengthy upfront training and individual authorization of each provider (such as Washington States’s now-sunset Limited Legal License Technician program)⁵ will be harder to scale than programs that authorize supervising organizations to iteratively train and insure the competence of providers (such as Alaska’s Community Justice Worker program).⁶ Barriers to replication are “what is required to replicate the program with fidelity to its original design and implementation.” (P. 1531.) For instance, state-level programs are harder to replicate than federal and Tribal programs due to state variations in unauthorized practice of law (UPL) regulation and rulemaking.

Barriers to learning are “constraints on experimentation and producing new knowledge.” (P. 1531.) Burnett and Sandefur emphasize the potential for community legal assistance programs to increase “people’s capability to engage with their own law” and argue that community learning should be a central goal of such programs. (P. 1528.) Regulators can encourage learning by authorizing experimentation by supervising organizations and requiring data collection and assessment as a component of program design. Here again, delegating authority to supervising organizations can increase scalability by allowing adaptation and expansion based on ongoing assessment.

Barriers to funding are what they sound like. As Burnett and Sandefur acknowledge:

Nearly all civil legal services programs and models face funding barriers, particularly in contexts such as the United States, where state and federal government funding for lawyer-based legal aid is inadequate to support programs of sufficient size to meet actual needs. Moreover, the traditional approach by philanthropic donors is often fragmented and project specific and relatively short-term rather than in the form of long-term operational support. (P. 1534.)

But while all efforts to expand legal assistance face funding constraints, Burnett and Sandefur identify three promising funding models that have emerged. The first is to allow supervising organizations to charge nominal fees (as is allowed by statute in the immigration context).⁷ Some programs that prohibit providers from receiving direct compensation from clients nevertheless allow compensation to supervising agencies (such as Delaware’s Qualified Tenant Advocate program).⁸ Another promising model is cross-subsidization, whereby justice workers are embedded in and paid by community organizations rather than the supervising organization (such as in Arizona’s Domestic Violence Legal Advocates program).⁹ A final model is to allow justice workers to work for profit and charge clients directly (as is allowed under the supervision of the Utah Regulatory Sandbox).¹⁰ Though there are obvious limits to a for-profit model for serving low-income clients and communities, some types of limited legal services may be marketable at low cost.¹¹

The final criterion in Burnett and Sandefur’s evaluative framework is sustainability, which refers to program “durability and resilience.” (P. 1536.) They identify four components of sustainability: diversity of funding streams; staff recruitment and retention; community engagement; and political support from the bar, legislatures, courts, and the public. Funding, community engagement, and political support are relatively self-explanatory and only partly within designers’ control, though Burnett and Sandefur point to several strategies that specific programs have used with success.

Staff recruitment and retention, however, are critical components for scaling and sustaining lay legal assistance programs and “have not historically been prioritized in program design and development.” (P. 1538.) The process of seeking regulatory approval tends to be onerous and time consuming and focused on ensuring effectiveness. But regulatory approval is just the first step. Once a program is authorized, it must be staffed, which requires workforce development and training and attention to worker retention and wellbeing. As Burnett and Sandefur acknowledge, we have little data on effective strategies for justice worker recruitment and retention, except that it helps to keep entry barriers low. It is also important to ensure community representation among justice workers, ideally including during program design. Burnett and Sandefur identify the Alaska Community Justice Worker program as a model in both respects but note that more research is needed. Staffing is also a persistent problem in legal aid offices staffed by lawyers, especially in rural locations.¹²

Three broad takeaways emerge from among the many specific contributions of these complementary articles. The first is the importance of adopting user-centered measures of effectiveness that consider not only the quality of service but also the ease and timing of access and user trust in providers. Here the profession and regulators should build on established models for assessment in related professions such as health and social services. The second is the significant benefit of working through existing legal aid and community organizations to train and supervise justice workers, rather than regulating individual providers. The U.S. profession has resisted entity regulation of legal services, in contrast to most comparable countries, which limits the models for service delivery in both the for-profit and nonprofit sectors. Scaling civil legal assistance depends on rethinking our uniquely individualistic approach. Finally, staffing is a persistent challenge, and we need models for workforce development, both through collaboration with related providers and through community-based recruitment.

Taken together, these articles contribute significantly to defining shared criteria and measures for evaluating legal assistance programs and identifying an action agenda for providers and regulators working to improve access to justice.

1. See Michael Houlberg & Janet Drobinske, The Landscape of Allied Legal Professional Programs in the United States, Institute for the Advancement of the American Legal System (2022), (surveying the landscape).
2. See, e.g. ABF’s Access to Justice Researchers Join in $1 Million Grant from the National Science Foundation, Sep. 21, 2023, (announcing a $1 million research grant to study the Alaska Community Justice Worker Program).
3. See Elizabeth Chambliss, Evidence-Based Lawyer Regulation, 97 Wash. U. L. Rev. 297 (2019) (discussing the increasing legal and political pressure for evidence-based regulation).
4. See also Matthew Burnett & Rebecca L. Sandefur, Designing Just Solutions at Scale: Lawyerless Legal Services and Evidence-Based Regulation, 19 Direito Publico 102 (2022) (discussing the importance of scalability in designing “lawyerless” models for legal assistance). “Around the world, billions of people lack access to justice, often because they cannot access help in resolving their justice issues. An important reason for this is that many access models rely centrally on lawyers, and such models simply cannot scale.”
5. See Wash. Admi. And. Prac. R. APR 28 (authorizing specially trained paralegals to offer limited fee-paid legal services). The program required a paralegal degree, 45 credit hours of law school coursework, the passage of three exams, purchase of malpractice insurance (which is not required of Washington state attorneys), and 1,500 hours of supervised practice by a licensed attorney. Id. The law school training was offered at one school, once a year, with limited capacity, “and outside the standard offerings eligible for financial aid.” Thomas Clarke & Rebecca L. Sandefur, Preliminary Evaluation of the Washington State Limited License Legal Technician Program, Mar. 2017. (Pp. 6-8.)
6. See Alaska Bar. R. 43.5 (authorizing a broad waiver of UPL restrictions for the Alaska Legal Services Corporation (ALSC), to train and supervise Community Justice Workers to offer limited legal service). The waiver allows ALSC to expand the services provided as the program develops, in partnership with the Alaska Native Tribal Health Consortium (ANTHC) and the Alaska Pacific University (APU). N.Y.U. Ctr. on Int’l Cooperation, The Native Health Partners Ensuring Justice for People in Alaska, MEDIUM (Oct. 2, 2020).
7. See 8 CFR §292.2(a)(l) (allowing recognized organizations to charge nominal fees to help supplement other sources of funding).
8. See Delaware Supreme Court Rule 57.1; Delaware Senate Bill Qualified Tenant Advocates; (authorizing trained lay advocates to represent tenants in eviction cases).
9. See Arizona Supreme Court Administrative Order No. 2020-88 (authorizing domestic violence advocates employed by the Emerge! Center Against Domestic Abuse to provide limited legal services to domestic violence survivors); Arizona Supreme Court Administrative Order No. 2023-21. (authorizing the expansion of the program to include other community-based organizations).
10. Our History, Utah Off. of Legal Servs. Innovation, (last visited June 24, 2024) (allowing both for-profit and not-for-profit organizations to seek waivers of UPL prohibitions, subject to a risk assessment of possible harm to consumers and ongoing monitoring).
11. See Elizabeth Chambliss, Marketing Legal Assistance, 148 Daedalus 98 (2019) (urging providers to design flat-fee, standardized products targeted to consumers’ discrete legal needs).
12. See, e.g., Damian Dominguez, S.C. Legal Services Closing its Greenwood office, Index-Journal, Jun. 27, 2024, (reporting that South Carolina Legal Services, the only state-wide legal aid provider, was forced to close its Greenwood office due to staffing shortages).