In order to effectively clinically reason, we need to be able to critique the evidence. I want to be clear from the start – I’m not here to sledge any authors or specific papers, so I’ll just use hypothetical examples throughout. But what I want to try and do is simplify the ability to critique research for those people who maybe aren’t comfortable doing so.
A few recent discussions with colleagues and MSc students at University prompted me to write this blog. I’m not a researcher and I’m certainly not a statistician. My wife just throws more than 3 sums at me to convince me I owe her money. Numbers fry my brain. But, that shouldn’t put me off being able to critique a paper in a constructive way.
Critical Comment #1: Can I understand why they’ve used this Methodology?
For an author to create a robust methodology, there has to be the existing literature available in the first place to support their design. We place a great deal of trust in authors that they have researched their methodology appropriately -the tests they use are validated, there’s evidence behind their outcomes, a clear rationale for their intervention. But have they made all of these clear? You can see already how we can create a peeled onion effect, whereby we could (if social lives weren’t an issue) trace back all of the references for outcomes measures and tests.
I feel a great deal of sympathy for authors here, because in some cases they cant win. Authors are torn due to previously limited research, to which they need to reference their proposed methodology in order to be considered robust.
Lets use something that’s not contentious, I don’t know…? Massage. No one has established an appropriate and valid duration. Neither have they determined best technique, and so on – so a great deal of literature these days will standardise their methodology to an arbitrary figure, often 2 minutes per technique. Where has this come from? For those who do use massage as part of their practice – when do you time a duration for techniques? Surely its individual, dependent on the therapist, the treatment outcomes and goals etc – but any paper that justified their methodology on something that is extremely subjective like clinicians experience would get slated!
I’m sure this will get shot down monumentally, but personally I would commend a study brave enough to use an experienced clinician and trust their clinical knowledge & autonomy. Let them use an intervention they use routinely and daily and allow for creative freedom and individual needs. We constantly bang on about treatments being individual, so lets put our money where our mouth is. I’ve used massage here, but the same could be applied for a lot of interventions – types, techniques, durations. If they haven’t been validated historically, how can we be assured about results from this current paper we’re critiquing?
It’s another argument for another time – but do we need to go back to basics with some interventions and learn more about them before we critique and dismiss them? Rather than compare intervention vs no intervention, should we compare the same intervention but with different goal posts first?
I’ve used massage here but that’s not my point, its the methodology I’m trying to emphasise.
- Is it a fair comparison between interventions?
- Does it even need a control?
Critical comment #2: Is there an appropriate population used for the research question?
We have to remember that any outcome or clinical relevance from a study can only be applied to the population that they used within that study. Can we assume that a new training program implemented with recreational athletes will have the same benefits with elite athletes? It’s impossible for authors to give us huge details about population because of their limited word count – but we need to make some educated guesses regarding the outcomes. The benefits of an eccentric intervention for an elite group of footballers doesn’t mean we can start Sunday league players or even semi-pro players on the same intervention at the same intensity or volume.
Take the findings and apply them to your clinical practice & patient exposure. Would this intervention fit with your athletes current schedule or level of conditioning?
Flip that around and consider that a study using a lay population may find huge benefits from an intervention – but is it just an accelerated learning curve that wouldn’t impact an elite athlete in the same way? Exposure to something completely new will have bigger consequences and effects.
Critical comment #3: The dreaded stats! Or am I just being Mean? Probably (<0.05)
I’ve already said, I’m no statistician. The critique that can be applied with some understanding of these stats processes is incredible and I am in awe of people that can do this. But there are some simple points to consider when looking through analysis and results of papers. The first thing to consider, does the presented data tell you what you need to know? Go back to secondary school maths with Mean, Median (and Mode):
We want to investigate how many hops a subject can manage after ankle mobilisations (assuming we had no other variables like fatigue etc). Their pre-test scores are around 50. During assessment they record the following scores (40, 51, 45, 52, 100), one time they have blinder, recording 100 hops. A mean score would suggest that the effect of mobilisations increased their pre-intervention scores from 50 to 57.6, this sounds quite impressive. A median score used in this example would tell us that aside from one outlier, their post-intervention scores didn’t change too much (51). In this case, we want to know for definite whether or not our mobilisations have allowed this subject to hop better – they have a world championships in hopping coming up. If the data is clearly presented, we may be able to work this out ourselves. But I’m lazy – I’ve got 30minutes over coffee to read an article, I want to read their results and discussions and hope that this leg work has been done for me.
Now an author wanting to get a publication is always going to present the data with greatest impact – in this case the mean. That’s fine, but its worth checking the number of scores recorded. The greater the amount of data, the more accurate a mean will be. But less subjects or less tests would always be worth double checking the data.
“If you can’t explain it simply, you don’t understand it” Albert Einstein
This brings us nicely onto probability. After writing this blog draft, I was shown this brilliant lecture by Rod Whiteley (Here) who understands this much more than me! (See above quote). It must be en vogue because the editorial in Physical Therapy in Sport this month disucsses P-Value also (Here). But what I do understand about P-Values is to always ask.. “So what?” So its statistically significant, but is it clinically relevant?
Again, another hypothetical study. We investigate the use of weighted squats to increase knee flexion. We find that by squat 1.5x body weight can significantly increase knee flexion (P<0.001). That significant difference is 3 degrees. Is that going to make your practice better? In some cases it may do! Achieving a few degrees in smaller joints with less room to play with, or perhaps post-op TKR and we just need a few more degrees to allow this patient to safely negotiate stairs – if they cant do stairs I’m not sure I would get them doing 1.5x BW squats though, which takes us back to our population critique.
Hopefully you have watched the Rod Whiteley lecture by now, so you can see where non-significant data can be very clinically relevant. It does make me wonder how much we have thrown out or dismissed that could be very beneficial.
Critical Comment #4: The Conclusion
So we have 30 minutes to quickly search for a paper, read the abstract and decide to read the article. I’ll hold my hands up to skimming the vast majority of a paper just to get to the conclusion. Not good practice though. Its worth checking who the author is, have they published on this topic before? What is their motivation? Most people will publish something that they either strongly believe in, or don’t believe at all. We’ve already discussed how its easy to manipulate stats, so if I strongly want to prove something works, given enough data & appropriate stats I could probably could. This sounds incredibly synical, but it should be a question you ask. If the conclusion is strong despite some variable results, bear it in mind.
Our Conclusion:
“Its actually quite exciting, what you know now will probably change”
So can we believe anything that’s published? Yes. We can & We need to. Otherwise we stand still. Being critical is not the same as disagreeing or dismissing something. It just shows us where there are gaps and where can start investigating next. It’s actually quite exciting, what you know now will probably change. Something you don’t understand now, we will probably find out in the future. But taking a single paper and changing our practice based on that is a bit drastic. We need to consider the body of literature, read articles that challenge accepted beliefs and make our own decision. The beauty of sports medicine is there are no recipes. Where possible the literature should challenge our thinking and keep us evolving, but it doesn’t always restrict us to guidelines and protocols. We are lucky enough to be autonomous in our treatment and our exercise prescription and we should celebrate that. Ask 3 respected conditioning coaches to create a program for one athlete with a specific goal and see how diverse they are. Thats what sets us apart from each other and makes us individual therapists and coaches.
Take home points:
- Check the methodology – are you happy with what they are investigating & how they do so? It is perfectly acceptable to disagree!
- Does the population used apply to what you’re looking to take from the paper? You are reading this paper for some reason – hopefully to re-inforce / change your practice. Do the female college basketball players used in this study apply to your clinical caseload?
- Don’t accept or dismiss a paper purely on its P-Value.
- Has the author based their opinion purely on the P-Value? Check! Don’t just accept their conclusion. This is their entitlement but its their interpretation of the stats.
#PrayForAuthors: They do face a fight between getting something published, and in doing so making their study conform to previously accepted literature but perhaps deviating away from what the masses actually practice in clinic. The lasting question I will leave you with; considering the points made in this blog and the discussion by Rod Whiteley – where does that leave systematic reviews? I have my own thoughts 😉 Let us know yours.
Yours in Sport
Sam
Plinths and Platforms 