MAPS: Re: MDMA Memory Study, December 1998

[David <dhe@xxxxxxxxx>]

On Mon, 4 Jan 1999, Per "Zike!" Carlbring went:

> Quoted from the article:
>  "RESULTS: Greater use of MDMA (total milligrams per month) was
> associated with greater impairment in immediate verbal memory (p <
> 0.02) and delayed visual memory (p < 0.06). "
> 
> The p-level 0.06 is not, I repeate NOT, to be considered as a
> statistically significant result! I am amazed! One should always set
> the alpha-level ("p") to 0.01 or 0.05. (Clark-Carter, 1997)

I don't think I'd take that approach toward criticizing the article.
The .05 level of significance is arbitrary; with only 24 participants
per group, it would be almost perverse to ignore a p<.06.  As you
yourself suggest, effect size is what's important here.

Now, what's vexing about the article is that it appears to contain
absolutely no descriptive data (e.g. group means and SDs on the memory
tests)--just the results of regression analyses on the z-scores.  By
definition, z-scores have a mean of 0 and a standard deviation of 1;
in this article, the authors derive their z-scores from the scores of
the non-MDMA control group.  But they never report the z-scores of the
MDMA group!  They just plunge right into regression analyses to see
what predicts 'em.

For example, the aforementioned "p<.06" comes from a regression of
delayed-visual-memory z-scores on monthly dose of MDMA (controlling
for vocabulary score and gender), for which the authors report a "beta
coefficient" of -0.0016.  I'm assuming that this is a standardized
beta.  Therefore, if I learned this stuff correctly, it means that
when dose-per-month increases by one standard deviation, delayed-
visual-memory z-scores decrease by .0016 standard deviations (which,
by the definition of a z-score, would be .0016).

However, I don't know what it means if dose-per-month increases by one
standard deviation--because the mean and SD of dose-per-month are not
reported!  Only the median and the range are reported.

Perhaps -0.0016 is _not_ a standardized beta, but rather a raw
regression coefficient.  In that case,if I learned this stuff
correctly, it means that every time dose-per-month increases by one
milligram, delayed-visual-memory z-scores decrease by 0.16%.  I still
don't know what that means, because there are no memory scores to look
at--not z, not raw, not anything.  I can't tell how any of the
participants did.

The entire Results secion is like that.

I see that our own Kit Bonson was thanked by the authors for comments.
Kit, can you de-confuse me about this article?

--David Epstein
  dhe@xxxxxxxxx
  the farthest thing from a statistician


------------------
MAPS-Forum@xxxxxxxx, a member service of the Multidisciplinary Association
for Psychedelic Studies (to become a member, see www.maps.org/memsub.html).
To [un]subscribe, email the message text,
[un]subscribe maps-forum youraddress to majordomo@xxxxxxxx