Recruitment shock
3.6% response rate? Shocking! For our new feasibility study, we sent over 200 invitations to primary care doctors in Ireland and the invitees sent us back a very strong signal. “We are not interested”, or “we are too busy”, or “we don’t have enough eligible patients”? Whatever the reason, the message remained the same: No, thanks.
The primary objective of our study, as for most feasibility studies, is to estimate numbers needed for a definitive trial. We want to know how many people should be invited into the study; of those, how many should be randomized; of those, how many will stay until the end. Right from the beginning, we were faced with a question whether we can recruit enough people for a fully-powered experiment.
Statistical power
Power in research experiments is about finding the truth. Experimenters want to know whether their drugs or treatments work. If the drug or treatment works and they give it to a group of people, some of them will improve, some won’t. There’s a lot of chance and uncertainty in any drug or treatment administration. If we want to know the truth beyond the effects of chance, we need to give the drug or treatment to the right number of people. There’s a formula for it, known to most statisticians. It depends on many things, like the size of the improvement that you want to observe in the treated group, or other confounding factors. The higher power in a study, the more likely it says true (see, e.g., Dr Paul D Ellis’, PhD site here).
A rule of thumb says that the more people are in the study, the higher the chances of finding a meaningful impact of the intervention. Common sense also tells us that the more people in the trial, the more representative they are of the whole population – the more confidence you can be that your results apply to all; except for Martians – unless you really want to study Martian citizenship.
Solution
The easiest would be to call some friends, doctors, and ask for a favor. This should work, but it’s not really scientific. Or you can shut down the study and conclude that it’s not feasible. Or you can do the study with the small number of interested participants. Or you can send another mailshot, a reminder, to all – sometimes that can help.
Fidelity questions
Clinical trials use elaborate methods to make sure that everybody does the exact thing as they planned. Measuring treatment fidelity is checking the agreement between study plan and practice. Some health problems require complex changes. How to measure fidelity in trials of complex interventions? Here are some ideas for fidelity checking.
The National Institutes of Health established a workgroup for treatment fidelity, as part of their behaviour change consortium (1999). They surveyed each centre in the consortium to find out which fidelity measures they use in trials. The workgroup recommendations span five areas: study design, training providers, delivery of treatment, receipt of treatment and enactment of treatment skills. They are useful for investigators who want to measure and improve their treatment fidelity. The key areas for our study are design, training, delivery and receipt.
Fidelity in our PINTA study
Our feasibility study has several aims. The first is to estimate parameters for a fully powered clinical trial. Secondly, we also want to know whether our intervention works. As a complex intervention, it targets multiple levels – doctor and patient level. We hope to improve doctors’ practices and patients’ health behaviour. Intervention fidelity in a multi-level study means adhering to different guidelines and processes. Our trainers must deliver uniform training to all learners groups. The doctors must provide consistent interventions to all patients in the intervention group.
Availability of personal portable audio recorders, e.g. smartphones, provides new and exciting opportunities for fidelity checking, but it raises some ethical issues. Doctors and other interventionists can easily record their consultations with patients and email them to researchers for fidelity checking, but email is not safe.
To avoid the potential confidentiality breach, the researchers can ring the doctors, give them a one-sentence brief and ask them what would they respond should this patient appear in their next appointment. Recording such phone calls is not a technical or ethical problem; it is not without limitations, though. Telephonic consultation with researcher in the role of patient does not reflect real life consultations and, as such, cannot be an accurate skills check. Doctors may not want to be called and recorded for quality assurance purposes, even if it’s anonymous and does not affect their income or professional standing.
When designing measures to improve treatment fidelity in our study, we have to consider how they will be perceived by our participants and providers. These are the strategies for monitoring and improving treatment fidelity that we plan to use:
Design:
-
Guidelines for primary care providers to manage problem alcohol use among problem drug users
- Scripted curriculum for the group training of providers
Training:
-
Booster session (practice visits) to prevent drift in provider skills
- Access of providers to research staff for questions about the intervention
Delivery:
-
Instructional video of patient–doctor interaction to standardize the delivery
- Cards with examples of standard drinks and scripted responses – to standardize the delivery
- Question about patient scenario in follow-up questionnaires (telephone contact)
Receipt:
-
SBIRT checklist for providers (process measure)
- Pre- and post training test (knowledge measure)
- Patient follow-up questionnaire will check whether each component of the intervention was delivered
Measuring fidelity in trials of complex interventions is important. It is not technically demanding. Ultimately this becomes a question of personal development and credibility – willingness to have one’s skills analysed and improved is the basis of reflective practice.
Saying bye slowly makes parting easier
Last days of my INVEST fellowship
Visiting research scholars make new friends quickly and parting is not always easy for them. I said bye in Portland (OR) five times:
First, I said bye to my writing group. This was my second group in the last 15 weeks. The first, 10-week course of prompt-based writing was a birthday gift from my wife. I enjoyed the first course so much that I decided to go for a second round. The new beginnings were difficult, because we had a new group and group dynamics; dynamics matters most in writing groups. By the 3rd-4thmeeting, the group juice started to flow and we shared more and more feedback on our writings. Parting with the second group wasn’t easy, but much smoother thanks to my experience with the first group; I felt I belong there.
Second, I said bye to the members of the Western States Node from the Clinical Trials Network. The network has 13 nodes funded by the National Institute for Drug Abuse (NIDA) to conduct clinical trials in addiction science. From the very beginning of my fellowship, I have attended weekly meetings of the team – around 20 in total. Marie made delicious cookies and Lynn gave me clock made of bike parts by a Portland artist. This was a well-chosen gift, because I cycled around Portland every day and really enjoyed it.
Third, I said bye to my colleagues from the Department of Public Health at the Oregon Health and Sciences University (OHSU). The lunch invite went to all faculty and staff, but I was worried that no one would come. I was worried that if I left, nobody will care about it. When I came there, I saw many familiar faces. It felt good because it made me feel like I have an impact on people that I managed to make a connection in a short time. We had a BYO lunch on the lawn. The sky didn’t look like 88 Fahrenheit but the sun shone on us eventually. Some of us sat on a white blanket from the ED which used to warm up patients as they came to ED. Many people showed up, including my mentor Dr Dennis McCartywith his wife; Dennis commissioned cookies and Sarah baked them.
In the evening, I said bye to the three of my best friends and neighbours plus their dog – Sonic. We stayed up late, talked, ate and listened to great music. Sonic honoured us with two carpet pees which destined him into his kennel for the rest of the night.
Fourth, I said bye to my dentist. Seth was a 3rd year dentistry student at OHSU School of Dentistry and helped me through many long visits. He reminded me about my appointments every Sunday night. Seth called me on Friday evenings and when he didn’t get an answer, he called back on Sunday night. He even called me when he had a cancellation to check if I had time to get some work done. When we parted, he pushed a bag full of toothbrushes and toothpastes into my hand; so that I take care of myself and my teeth. We had a lot in common, especially the taste for adventure. I surprised my wife with a hot balloon ride for her birthday last Thursday and he treated his wife with the same ride for their 3rd anniversary.
Fifth, I said bye to my mentor, Dr Dennis McCarty. When I arrived to the department that morning, it was pretty empty and my heart sank because I haven’t had a chance to say bye to Dennis. But he came later. Dennis helped me to improve and expand my writing. I’ve read four books on writing, borrowed from him, during the my fellowship. I’ve never read so much about writing in my life. Dennis introduced me to science writers, e.g. Atul Gawandeor Carol Cruzan Morton, science writing, e.g. JRF publications, and science writing competition – the Wellcome trust prize. We met at the career crossroads – an emerging science apprentice and a seasoned mentor. He taught me that research project management is unlike any other research skills: you don’t learn these things by reading books or in the classroom, but through the apprenticeship. He was not only my mentor, but at times, acted like my guide, counsellor, teacher, proof reader, father and friend.
My point here – that saying bye slowly makes parting easier – should interest most visiting research scholars. Beyond this limited audience, however, my point should speak to anyone who faces parting with many good friends.
Honor pot: testing doctors’ drug counselling skills in a new pilot study in Ireland
In our new new paper, we outline plans for doing a study which should tell us whether doctors and agonist patients accept psychological interventions as means of curbing alcohol in primary care; it should also tell us whether we can do more research on this topic in Ireland. Access the full protocol here http://www.researchprotocols.org/2013/2/e26/
For some people, publishing research protocols is not fun because of two reasons:
- everybody knows what you’re doing
- you have to do what you said – everybody knows now.
However tough for researchers, these two reasons make publicly available research protocols the best way to achieve transparency in research. Transparent research is in line with ethical principles of research conduct and makes an honorable contribution to the scientific knowledge – to the honor pot. Together with accountability, it should be the core pillar of scientific discovery.
If these safeguards fail, we may see more instances of academic fraud and data falsification, such as Diederik Stapels’. The social psychology community has been embarassed by the revelation that Diederik Stapels made up the data for his papers. The NY Times link provides a detailed analysis of the Stapels and his academic fraud.
Helping agonist patients with alcohol problems: A NEW guide for primary care staff
What should doctors do differently when screening for alcohol use and delivering brief interventions for agonist patients in primary care? General principles remain the same like for other people, but:
-
the screening and treatment processes should be more systematic and proactive in all problem drug users, especially in those with concurrent chronic illnesses or psychiatric co-morbidity,
-
lower thresholds should be applied for both identification and intervention of problem alcohol use and referral to specialist services,
-
special skills and specialist supervision is required if managing persistent/dependent alcohol use in primary care.
To read the full guideline, click here (http://link.springer.com/article/10.1007/s11845-013-0982-2#)