Trust in data

One of the risks I take in writing this column is opening myself up to ridicule. In order to set the scene for my thoughts, I may divulge things better kept under wraps. This is one of those occasions. I announce to you, with some trepidation, that I have rediscovered full-fat milk! After years of embracing semi-skimmed in an effort to keep healthy and resisting the pressure to go the whole hog and take the next step of drinking the white water otherwise referred to as skimmed milk, I have rediscovered my youthful joy in a cold glass of full-cream milk, especially after exercise. The health warnings about fat have been modified in recent years but the message is deeply ingrained – so much so that, whilst I might reminisce about putting full-fat milk or, better still, the cream from the top of the milk bottle, into my bowl of cereal, I persist with semi-skimmed. Of course, sugar is the supervillain nowadays so I should refrain from cereal, but old habits die hard. There is probably a study somewhere that shows that even reminiscing about full-fat milk is bad for you!

Last month, I mentioned how a study about butter funded by the butter industry had reported that butter is bad for you. The context illustrated that we do not trust research paid for by a party with a vested interest in a particular result. Sometimes research also contradicts the advice in previous studies: red wine is good for you; red wine is bad for you. Now there is even more reason for doubt, after the Centre for Open Science at the University of Virginia undertook a reproducibility project. Five landmark studies in cancer biology were repeated but the results of only two of the five were able to be reproduced. Two were inconclusive and one was a total failure.

The news referred to a reproducibility crisis: science is built on replication and if we cannot reproduce results, how do we know if they are true? Can we trust them? The problem lies in the prevailing competitive culture. Researchers and publications are rewarded for presenting results that are novel and have impact. Unambiguous findings reap the greatest rewards but this can lead to selective reporting and a failure to disclose results in full. The kudos and monetary reward may be a lot lower in NDT research and development than in other more high-profile scientific disciplines, but similar pressures will apply. Moreover, the safety-related consequences of relying on reported information that doesn’t tell the full story can be serious. One piece of information I was unable to squeeze in last month, following the butter revelation, was the fact that we can place more trust in the papers in Insight because they are subject to peer review prior to being accepted for publication. This review is undertaken by a volunteer army – or perhaps more of a grouping – of technical experts. This type of review is standard practice in science and engineering and, although it is a valuable and necessary filter, it obviously falls short of actually reproducing the experiments.

Papers presented at larger conferences are reviewed at the submission stage and selected based on their abstracts. At other events there may not be any vetting of the submissions. Is this a problem? Maybe not, but if we do not know the full story how can we make a judgement? In the spirit of continuous improvement, we can take inspiration from other sectors. Those readers with children at school may have noticed that, when work is marked, there is now an initiative to report: What went well? What can be improved? Would it be useful to have a similar initiative in NDT? We could ask that authors add an appendix to their papers, describing the challenges that they faced and highlighting further work that could enhance the applicability of their results. This would be analogous to the ‘how we made this film’ clip that is appended to many documentaries on TV.

Another idea could be to reward the best papers in a journal or at a conference with a special prize: an independent validation of the results. Previously reported work is a key source of information for inspection qualification. It is a way of avoiding the unnecessary cost of reinventing the wheel. In the absence of being able to easily judge the provenance of important studies, maybe it is necessary to publicly reproduce their results at least once to achieve the required confidence.

Please note that the views expressed in this column are the author’s own personal ramblings for the purpose of encouraging discussion within the NDT Newspaper. They do not represent the views of Amec Foster Wheeler or the HSE who funded the PANI projects.

Letters can be mailed to The Editor, NDT News, Midsummer House, Riverside Way, Bedford Road, Northampton NN1 5NX. Fax: +44 (0)1604 438300; Email: ndtnews@bindt.org or email Bernard McGrath direct at bernard.mcgrath@amecfw.com

Comments by members

Comment by David Bilsland
Blairgowrie, United Kingdom

Date 20/04/2017

Being that we all have to re-certifiy every five/ten years I think it would be fair (if not happening already) that all level III's working in training schools should, every five years, go out and do NDT work offshore/pipeline/onshore plant to validate their skill re their discipline. This seems to be common sense and fair play.
Re the above post...I never left full fat milk and butter...the real stuff is always better!

Kind regards

<< Back