Validation of an OPAC delirium tool

In this post I wish to explore validation of one of the tools recommended by Healthcare Improvement Scotland to identify delirium in acute hospital.

The website for the 4AT can be accessed here. It states that:

4AT validated

Looking at these two published studies in turn:

053Tools

The following caveats are included in this validation study:

  1. One of the  main features of the 4AT test is that “no special training is required”. In this study 4AT assessments were “performed by experienced physicians, though no specific training in the 4AT was given”. The authors conclude that: “Further research is needed to assess the ease of use of 4AT among other professional groups of varying levels of seniority.”
  2. The authors also state that this study “did not assess inter-rater reliability for the 4AT or the reference standard assessment”
  3. This study was not set in A& E or hospital front-door settings.
  4. The authors of this study conclude that “because of insufficient power, we were not able to analyse the characteristics of misclassified (false negative and false positive) patients.”
  5. The authors of this study advise that “future studies in larger populations and other centres should further assess its performance, including the determination of whether detection of delirium using the 4AT may improve the clinical outcomes of patients.”

It seems very clear to me that the authors of this study have recognised that further assessment of the 4AT is required before its use can be considered to be reliable in populations other than that studied by the authors.

Here follows the second published study quoted as evidence of validation of the 4AT test:

054Tools

This study is set specifically in a Stroke Unit and thus its conclusions cannot be generalised to all acute admissions.

It is therefore welcome to see that a major study is being undertaken to further evaluate the 4AT test:

4AT validated UK Gov

Here are some extracts from the study protocol:

055Tools       051Tools

This study is not yet published. The protocol states that: 4AT further research

This research study has had ethical approval and is rigorous in its approach to patient consent. Obviously a research study includes different ethical dimensions when it comes to consent but, as I have reasoned in my last post, the use of a test that still remains to be fully evaluated also requires that ethical rigour.

informed consent

The study protocol outlines the hope that:

NICE delirium 2

However the study protocol also states that:

knowledge-mobilisation-4at-3

Healthcare Improvement Scotland, in their recent reports state that “OPAC tools work and are working in hospitals across Scotland”. One of these tools is the 4AT test.

In summary there must always come a stage where a research tool is evaluated in the ‘real world’. My concern is that Healthcare Improvement Scotland is encouraging the use of the 4AT in hospitals across Scotland without making it clear that it is still under evaluation. Given that “compliance” with the use of “OPAC tools” is being used as a measure of quality of care by Healthcare Improvement Scotland, it would seem justified to express concern about this.

Professor David Oliver recently had an opinion piece published 
in his regular BMJ column; Delirium Matters (24th May 2016). 
I wrote to Professor Oliver by e-mail to say that I agreed with most 
of what he said but that I was not as convinced as he was 
that the 4AT has been validated.

Professor David Oliver replied that it "beggared belief" 
that I would "nit pick" about this.

I replied:

"In terms of the word “validated”: this has a very clear 
scientific definition. My reading of the science so far is 
that the 4AT tool has not been validated either for 
screening or detection. 

My concern is a general one: that “simple”, “quick” and 
“no specialist training required” tools have the potential 
for harm when over-relied upon and that they may give 
the impression that a very complex condition/state 
is more simple and better understood than it really is."

 

 

 

One thought on “Validation of an OPAC delirium tool

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s