Philobiblon: But where's the evidence?

Monday, November 28, 2005

But where's the evidence?

Perhaps I've become more critical over the years, or perhaps newspapers have got worse - probably it is a bit of both - but increasingly I read newspaper articles, get to the end, and ask, what? What evidence have you got to support the contention you've made here? Why is the paragraph that debunks the whole story (if it is there at all) buried in the depths of the article? (Well I know why, but why was the reporter allowed to get away with it?)

Two examples. They're from the Guardian and Observer, probably the least worst in their respective markets in this regard, but nonetheless, there they are, on their websites.

Case One

Today there's a piece about teenage self-harm. The headline reads: "Sense of failure: the scale of teenage self-harm. Study shows one in five girls has wounded herself. 'Must-have' culture brings feelings of inadequacy". And that's probably all a majority of readers will look at, and go away with the common message: "society is going to hell".

But I read on. And an alarm flag went up in paragraph two. "A survey published today by The Priory, which specialises in treating mental health problems and addictions..." A hunt through the rest of the story provides no more details about who conducted the survey - a university department, or even survey company, or an employee of The Priory. Mmmmm ... a survey company or employee finds that there's much more need for services of the company that commissioned it to do the survey. (Because had there been a university it would surely have been mentioned to bolster the story.) And this is the whole foundation for the story. Bit of a worry ...

Next: Applied to the general population, survey means more than 1 million British adolescents have considered self-harm and more than 800,000 (13%) actually inflicted injuries on themselves. But was this survey representative of the general population? No data provided, but I have a funny suspicion that it will have been conducted in London and environs, probably amid middle-class kids - the ones likely to end up at The Priory ... hardly representative of a whole country. Interesting too, that there is no attribution for who is making the extrapolation.

... A national inquiry into the prevalence of self-harm among British teenagers by the Mental Health Foundation and the Camelot Foundation is due to report next year. That, hopefully, will have proper methodology, and will be the study this story should be waiting for.

...According to Childline, the numbers of youngsters calling its helpline about self-harm has risen by 20% in the last 10 years, with a marked increase - 30% - this year. Has it increased services in that time, has it done more to encourage children to ring, has the issue got more publicity? Does this figure mean anything at all? (Except that the charity - probably with good intentions, wants more money?)

... According to Dr Griffiths, the increased reports of self-harm may also be a reflection of contemporary society and the media, with their emphasis on fame, celebrity and "instant gratification". I do like that "may".

Finally, the last paragraph .. According to The Priory, most self-harming is symbolic - typically involving small cuts that do not draw blood and are invisible to teachers and parents. The practice releases natural opioids which can be "incredibly addictive". So after we've all been having lurid images of wrists dripping blood and attempted suicide, we get small scratches. If, and it is a big if, we've read to the last paragraph.

So the one-sentence summary - Commercial company commissions survey that finds a greater need for its services. A bit weaker than the original, but more accurate.

Case 2
In the Observer, Why the have-it-all woman has decided she doesn't want it all. The sub-head reads: "As a new generation of mothers seeks to change the balance between work and home, Tessa Jowell calls for a debate on how we all live". And there's a "politics" section logo.

Paragraph two: "But now, the Having All It All generation are giving way to the Actually, I Don't Want It All - or at least, Not All At The Same Time generation. And their champion comes from a somewhat unusual quarter. The government's minister for women declares today that modern women are increasingly unwilling to bear the stress of trying to do everything at once - and calls on men to share more of the responsibilities at home."

So goes the whole story. The basic premise here is "we are responding to a fundamental change in what women want". But, wait a minute, where is the evidence that this has changed - maybe some statistics on workforce participation, some solid social science survey, hell, even a well-conducted "pop" survey?

None, nada, not a word. The whole story is built on a premise - a very large premise - that it makes no attempt to justify or back up.

But it is worse than that: More than half of British women are currently working in a job for which they are overqualified, often because domestic responsibilities leave them too little time or energy to pursue more senior positions. "Often" - what does that mean? I could equally say - and would say - "often" women are stalled in their jobs because of male prejudice and discrimination, "often" all workers, men and women, are stalled in their careers for all sorts of reasons - from their boss not liking them to their inability to move location because of their children's schooling ...

Now I don't want to pick on these two particular journalists - they are only cogs in the wheel, and the stories products of the huge pressure to produce great headlines. But such a pity those headlines so often have no solid foundation whatsoever, and yet these are what give readers their view of the world, that guides their votes and their actions.


Blogger Oscar Wildebeest said...

So, four out of five British girls (at least) are not self-harming. That sounds like good news. Wouldn't sell any newspapers, though (can't you just see the headline: "84% of British people say, 'we're doing all right, thanks very much'" - no, you can't, can you?).

11/28/2005 11:54:00 am  
Blogger Natalie Bennett said...

I don't know - has anyone ever tried? A sort of anti-Mail - "hey, things are going really pretty well", and here's the stories to prove it - would be an interesting concept to try.

11/28/2005 12:43:00 pm  
Blogger Ronnie Smartt said...

The Observer piece in particular has a familiar ring. One would not be at all surprised to read the opposite next week. The activity of journalists is not always to give us news, let alone the truth, but to have us going "Gosh" or "Have you seen this?" or "It really is a disgrace" or "You'ld think the Government...", and generally to stimulate conversation among usually taciturn people.

11/28/2005 12:54:00 pm  
Blogger Natalie Bennett said...

... but how many people believe it - and then act on it? Not necessarily because they're dumb, but because they are only giving half their attention or less.

11/28/2005 01:10:00 pm  
Anonymous Sharon said...

I'm losing count of the numbers of stories like this. Although the technique has almost certainly been around since the early 20th century; I seem to recall some good interwar examples in a book that should be compulsory reading in schools: D Huff's How to lie with statistics...

Anyway, after a fair bit of searching, I've come up with the report on which the newspaper story is based:

The Priory says the survey was nationwide and carried out by an independent company. I haven't had time to read the report yet: might be interesting to compare what it says to the newspaper stories (which are probably based just on the press release...).

But, I ask myself, why did I have to take time out digging around to find the link? It's high time the online versions of newspaper articles included direct links to external sources like this as standard (the BBC website does it to some extent, but the newspapers rarely seem to). This irritates me.

11/28/2005 02:33:00 pm  
Blogger Natalie Bennett said...

Thanks Sharon - good point. There's no reason why the Guardian couldn't provide the link - except I suspect that they'd fear they would lose readers to "the real thing".

Having a look at it, the report isn't, of course, as bad as the newspaper article, but it does run the alarmist line whenever possible.

I particularly liked the line about "adolescents who self-harmed were rare 30 years ago". I'd like to see the evidence for that - since I suspect almost no one was looking for it, there won't be any statistics.

I was a teen 20-25 years ago, and I knew a lot of my peers self-harmed, but I doubt their parents did, let alone their doctors. And they themselves probably wouldn't even have had a label for it.

11/28/2005 07:47:00 pm  

Post a Comment

Links to this post:

Create a Link

<< Home