In this post I wish to explore validation of one of the tools recommended by Healthcare Improvement Scotland to identify delirium in acute hospital.
The website for the 4AT can be accessed here. It states that:
Looking at these two published studies in turn:
Validation Study One:
The following caveats are included in this validation study:
- One of the main features of the 4AT test is that “no special training is required”. In this study 4AT assessments were “performed by experienced physicians, though no specific training in the 4AT was given”. The authors conclude that: “Further research is needed to assess the ease of use of 4AT among other professional groups of varying levels of seniority.”
- The authors also state that this study “did not assess inter-rater reliability for the 4AT or the reference standard assessment”
- This study was not set in A& E or hospital front-door settings.
- The authors of this study conclude that “because of insufficient power, we were not able to analyse the characteristics of misclassified (false negative and false positive) patients.”
- The authors of this study advise that “future studies in larger populations and other centres should further assess its performance, including the determination of whether detection of delirium using the 4AT may improve the clinical outcomes of patients.”
It seems very clear to me that the authors of this study have recognised that further assessment of the 4AT is required before its use can be considered to be reliable in populations other than that studied by the authors.
Validation Study Two:
Here follows the second published study quoted as evidence of validation of the 4AT test:
This study is set specifically in a Stroke Unit and thus its conclusions cannot be generalised to all acute admissions.
It is therefore welcome to see that a major study is being undertaken to further evaluate the 4AT test:
Here are some extracts from the study protocol:
This study is not yet published. The protocol states that:
This research study has had ethical approval and is rigorous in its approach to patient consent. Obviously a research study includes different ethical dimensions when it comes to consent but, as I have reasoned in my last post, the use of a test that still remains to be fully evaluated also requires that ethical rigour.
The study protocol outlines the hope that:
However the study protocol also states that:
Healthcare Improvement Scotland, in their recent reports state that “OPAC tools work and are working in hospitals across Scotland”. One of these tools is the 4AT test.
In summary there must always come a stage where a research tool is evaluated in the ‘real world’. My concern is that Healthcare Improvement Scotland is encouraging the use of the 4AT in hospitals across Scotland without making it clear that it is still under evaluation. Given that “compliance” with the use of “OPAC tools” is being used as a measure of quality of care by Healthcare Improvement Scotland, it would seem justified to express concern about this.
Professor David Oliver recently had an opinion piece published in his regular BMJ column; Delirium Matters (24th May 2016). I wrote to Professor Oliver by e-mail to say that I agreed with most of what he said but that I was not as convinced as he was that the 4AT has been validated. Professor David Oliver replied that it "beggared belief" that I would "nit pick" about this. I replied: "In terms of the word “validated”: this has a very clear scientific definition. My reading of the science so far is that the 4AT tool has not been validated either for screening or detection. My concern is a general one: that “simple”, “quick” and “no specialist training required” tools have the potential for harm when over-relied upon and that they may give the impression that a very complex condition/state is more simple and better understood than it really is."