A bid to establish a standard quality measure for C-L Psychiatry throughout the US has encountered pitfalls but also made progress—thanks to painstaking work behind the scenes by Academy task force and SIG members, endorsed by the ACLP Board.
Here we talk, first, with David Kroll, MD, chair of the Quality & Safety SIG, who has been writing an occasional business column for ACLP News; then with Thomas Heinrich, MD, FACLP, chair of the ACLP Benchmarking Task Force, which is tasked with developing benchmarks for C-L Psychiatry.
In a bid to establish a standard quality measure
In your last Business article for ACLP News you talked about the challenges of establishing a consistent way for C-L Psychiatry services to demonstrate their value, not least because there is no consensus among C-L psychiatrists on what should be measured. It sounds as though it’s been a frustrating experience to date?
“If it were easy, it would have been done a long time ago!” says Dr. Kroll. “But I would say that actually there has been some movement toward more of a consensus, at least that different measurements are different for important reasons.
“Metrics that directly pertain to the structure and operations of the C-L service itself will be useful for C-L Psychiatry service leaders who seek a benchmark against themselves, and to some extent against other services.
“But if we are going to use metrics to make a case to the hospital systems we work for that C-L Psychiatry services are a worthwhile investment (although most tertiary hospital systems see this by now, many hospital systems still don’t, sadly), we need to measure patient outcomes.
“And those patient outcomes need to matter, regardless of whether a psychiatry consult occurred—otherwise it will appear that an under-resourced C-L Psychiatry service that can handle only a small volume of consults does just as well as a service that is adequately staffed. In other words, we have to be willing to find that a C-L Psychiatry service with adequate resources might perform better than those that do not have adequate resources—if we’re ever going to make a compelling case that those resources are important.”
Last year you led a pilot in a bid to test a standardized quality measurement for C-L Psychiatry. What did you do? What did you learn from it?
“We brought together nine C-L Psychiatry service leaders from around the US to test the feasibility of applying different methods of gathering routine data, rather than try to create the standardized measurement tool of the future.
“These methods reviewed administrative data for service volume, conducted satisfaction surveys among consultees, recorded service timeliness based on time stamps, and audited charts for documentation elements. We learned a lot.
“First, it’s really hard for different services to collect the same metrics because they have different structures and different resources, especially the ability of electronic health records to capture and report data automatically.
“Second, the metrics that require the least amount of physical effort by staff were the easiest to get. That shouldn’t be surprising, but some of the leading measurement strategy guides for C-L Psychiatry services that exist in the literature require an enormous investment of time that I just don’t think is feasible for everyone.
“That’s not to say that the published protocols aren’t useful—they really help to inform the development of better and better models, but I don’t think they’re the end point.”
Even so, you say trying to establish a standardized quality measure to date has not been successful. What are the main reasons? Different institutional or state standards/practices? The wide variety of clinical services and settings in which ACLP members practice? Apathy about a complex subject that to some still seems something we shouldn’t have to be involved with?
“I think the most important reason is that so far the agencies that are leading national efforts to define and measure quality—like CMS, The Centers for Medicare & Medicaid Services—have other priorities.
“CLABSI (catheter-associated blood stream infections) rates reflect a readily identifiable problem to hospital systems that is based on lab results and gets better when its measurement is included in quality payment programs. But think about conditions where C-L Psychiatry services have the most ownership: delirium and suicide.
“Delirium measurement requires manual collection (typically, nurses administering the confusion assessment method over and over again), and it is not a part of routine practice everywhere. Meanwhile, we don’t know yet whether routinely measuring delirium onset or outcomes would lead to improvements because, to my knowledge, this question has never been studied. Suicide most commonly happens after hospital discharge, so most hospital record systems can’t keep track of it. We’d be relying on the states or municipalities to keep track of this, and public reporting of suicide isn’t reliable enough in a lot of places.
“And so, is there an ambivalence or apathy on the part of C-L Psychiatry services or hospital systems to measure quality? I don’t think that’s a fair question. Hospital systems have to prioritize what they’re required to measure. We’re still largely on our own with this project.”
A paper on the project you offered to journals for publication has been rejected. Do they give any feedback?
“They give plenty of feedback, and some of it’s even good. But an important difference between quality improvement studies and traditional research studies is that quality improvement studies need to be unequivocally successful in order to be interesting to the general public.
“The goal of a traditional research study is to ask a specific question and find an accurate answer to that question. It doesn’t matter if that answer is ‘yes’ or ‘no,’ so long as it’s supported by the study. The word for this is ‘equipoise’, which is the opposite of bias. The goal of QI is to solve the identified problem, which typically means tweaking the intervention over and over again until you get a resounding ‘yes.’
“I think the main reason we’ve had trouble getting our study published is that we didn’t reach that resounding ‘yes.’ We learned a lot, and we got a lot of things right by the end, but the original problem statement was that a standardized approach to measuring quality doesn’t exist yet, and the problem still remains.”
You’ve written in the Business column about how trying to develop a quality measure applicable to C-L Psychiatry services would be a big commitment for ACLP to undertake on its own. But, if that’s our goal, we wouldn’t need to do it alone. You’ve commented: “Typically, the groups that develop measures are large organizations that already have a major stake in health care quality, like the National Committee for Quality Assurance, but medical societies (for example, the American Psychiatric Association—or even the ACLP) can do so as well… It is a cumbersome process that takes years (and a lot of money). We could similarly partner with other organizations that have different resources and perhaps more experience in this area—if we wanted to, and if only we can make a compelling case to them for why it’s important.” ACLP’s Board has shown its enthusiasm for the work of the Academy’s Benchmarking Task Force. That would seem to be a first step to the Academy getting involved and creating a pathway towards mutual objectives?
“Definitely,” says Dr. Kroll. “I think a lot of great things have come out of the task force already. The first is that we’ve started to collect an inventory about what people have the resources to do and are already doing. The second is that the ACLP Board has moved towards a consensus about what it thinks is important. The ACLP leading the effort is an important step beyond individual service leaders running projects on their own.”
Among the taskforce’s goals is to develop a response to the UK’s PLAN guidelines (that have been supported by ACLP)? Could you sum up their content?
“The PLAN guidelines were developed by a consortium of liaison psychiatry services in the UK who wanted to develop benchmarks for what they consider an excellent C-L Psychiatry service worthy of accreditation within their network.
“The list of benchmarks is extensive, and meeting them requires applying for accreditation and undergoing an audit by the accrediting body. My understanding is that it’s somewhat akin to a US hospital seeking Magnet designation for excellence in nursing.
“The ACLP could choose to emulate this—to say that we want to define quality based on meeting a set of metrics that may, or may not, look a lot like the PLAN guidelines. The outcome would likely be a binary designation of ‘accredited’ or ‘not accredited’ and may, or may not, support the underlying goal of justifying a bigger investment in C-L Psychiatry.
“The applications and audits would likely require a lot of time (and probably money) on the part of C-L Psychiatry service leaders and the ACLP members charged with the program. But, even if we didn’t copy the PLAN manual exactly, I think it would be useful to pore over the benchmarks the PLAN authors lay out and decide which ones we agree with and which ones we don’t.”
The Benchmarking Task Force is also preparing to determine how C-L Psychiatry providers’ productivity and quality outcomes are measured. How will the Task Force look at these measures?
“The Task Force will be approaching this important topic from two angles,” says Dr. Heinrich. “First, we will be surveying ACLP membership to better qualify providers’ understanding of these important measures, as well as quantify actual benchmarks identified by C-L psychiatrists.
“Second, we are going to survey the leadership of a geographically representative sample of selected academic and community departments of psychiatry to further (and maybe better) define current productivity, staffing, and quality benchmarking expectations for inpatient and outpatient C-L Psychiatry services in the US. We are going to need membership’s help in identifying departments willing and able to share this valuable data.”
Initially, the task force is drafting process and outcome quality metrics for inpatient C-L Psychiatry services as well as for outpatient collaborative care locations?
“The goal of ACLP’s Business Benchmarking Task Force is to provide members with credible C- L Psychiatry-specific quality and productive benchmarks to aid in communication and negotiation with practice, department, and hospital leadership.
“However, the wide variety of clinical services and settings in which ACLP membership practices makes for a potentially overwhelming number of potential service- and setting-specific benchmarks. As a result, the task force will initially focus on establishing productivity, staffing, and quality benchmarks for inpatient C-L Psychiatry services, as well as outpatient collaborative care practices.
“Hopefully, knowledge of this data [from a member survey] will allow membership to more effectively lead in the development or maintenance of high-quality clinical programs.”
You have also explored various incentives to improve response rates to surveys—because past surveys on such issues have failed due to poor response rates?
“The ACLP leadership has been very supportive of the project since its inception over a year ago. The project is sponsored by the Board and it could not have made it this far without the support and engagement of a great many within the leadership and general membership of the ACLP. Thanks go to everyone who has been involved over the past year.
“A great example of the Board’s support has been their generous donation of a complimentary registration to this year’s annual meeting in San Diego. This recognition will be awarded to one randomly selected member who completes the member survey. As the survey will close after the annual meeting, the ACLP will reimburse the lucky winner the registration fee after the fact.”
Even with sufficient data, you’d expect difficulty in adapting diverse metric reporting systems into one database?
“As mentioned previously, there are many potential challenges in making the gathered data reliable and generalizable for the ACLP membership. In an attempt to address some of these challenges, the Task Force has developed two IRB-approved surveys and limited the initial benchmarks to inpatient C-L Psychiatry service and outpatient collaborative care models in both academic and non-academic settings.
“The two surveys attempt to gather their respective data from the sources most likely to ‘hold’ the requested information (ACLP members and the departments in which they work). It is recognized that the diversity of practice sites (inpatient, outpatient, academic, non-academic, etc.,), types of practice (proactive, reactive, collaborative care, collocated care, etc.,) and providers (physician, psychology, AAP, residents) makes gathering data that captures the work of all ACLP members difficult and will require further efforts to clarify benchmarks for the many variations of C-L Psychiatry.”
Is there a timeline for the membership survey as yet? Data is said to be due for distribution to the Academy membership during the next six months?
Yes. We look to launch the member survey later this fall and close it shortly after the ACLP meeting. We look forward to sharing the results of the survey with the ACLP membership soon afterwards. Please look for the survey in your email account later this fall and thanks in advance for completing the survey… Every single response counts!
“We will also need membership’s help in identifying departments and/or hospital systems willing to complete the institutional-level survey. This IRB-approved survey will be distributed after the New Year once an adequate number of geographically dispersed institutions are identified by membership.”