||"Our own rabbis experiments.|
Perhaps the most important class of experiments we have conducted
are repetitions of the famous rabbis experiment. For this purpose, we
engaged Simcha Emanuel, a specialist in rabbinical history at Tel-Aviv
University, as an independent consultant.
For the first experiment, Emanuel was informed which 32 rabbis appeared
on WRR's second list and asked to prepare names and appellations for each
of them. He had not seen WRR's lists and was asked not to consult them,
nor was he given any explicit guidance concerning which types of appellations
to include and how to spell them. Rather, he was asked to use his own
professional judgement to settle all issues. During his work he consulted
a second historian, David Assaf of Tel-Aviv University. As well as writing
names and appellations, Emanuel and Assaf commented on the accuracy of
the dates given by Margaliot (1962) and corrected some of them (as had
The result of this experiment was a list of names and appellations which
appears quite different from that of WRR. The least permutation rank of
P1 _ 4 was 0.233.
The same exercise was then carried out with a list of rabbis that had
not been used before, namely those whose entries in Margaliot's encyclopedia
occupy from 1 to 1.5 columns and for whom there is a date of birth or
death mentioned (except for those incorrectly included by WRR in their
second list). For these 26 rabbis, the least permutation rank of P1 _
4 was 0.404.
After the above two experiments were completed, we carried out the following
re-enactment of WRR's second experiment.
1. A list of rabbis
was drawn from Margaliot's encyclopedia by applying WRR's criteria for
their second list, while correcting the errors they made. Our list differed
from WRR's in dropping two rabbis and including three others. One rabbi
who fits the selection criteria could not be included because he appears
incorrectly in WRR's first list.
2. Emanuel was shown
the spelling rules and table of appellations for WRR's first list as they
first appeared in WRR (1986). He then compiled a parallel table of appellations
for our list of 33 rabbis, attempting to follow the rules and practices
of WRR's first list.
3. To mimic WRR's processing
of dates for their first list, we used the dates given by Margaliot except
in the cases where Emanuel either found an error or found an additional
date. In some cases Emanuel regarded a date as uncertain, in which case
we followed WRR's practice of leaving the date out. Overall, Emanuel changed
more of Margaliot's dates than WRR did.
4. The resulting list
of word pairs was processed using WRR's permutation test.
The result of applying WRR's permutation test was that the least permutation
rank of P1 _ 4 was an uninteresting 0.254.
There are some syntactic differences between Emanuel's list and WRR's
first list, namely that Emanuel was sparing in use of articles and sometimes
used a one-letter abbreviation for "Rabbi". We pointed out these
differences to Emanuel, who then made some changes to his list. Because
of our intervention, the new list cannot be said to be as a priori as
the original, but it is arguably closer to the practices of WRR's first
list. The new list gives permutation ranks of 0.154, 0.054, 0.089, and
0.017 for P1 _ 4, respectively. Applying the Bonferroni inequality as
in WRR94, we have an overall significance level of 0.066.
This negative result is all the more conclusive if we realize that our
experiment had some clear biases towards WRR's experiment. The definition
of the set of rabbis, the introduction of P3 and P4 (only P1 and P2 appeared
with the first list) and, most importantly, the definition of the permutation
test, were under WRR's control when they ran their second experiment and
were merely copied by us. Thus, we were vulnerable to any systematic bias
that existed in those decisions, as well as to the possibility that WRR
knew some examples from their second list earlier than acknowledged. We
can only partly compensate for these biases. Using only P1 and P2 changes
the overall result to 0.108. Using the permutation test of Diaconis (discussed
in Sections 3 and 4) rather than the test invented by WRR, the results
are even worse: 0.647 using the average and 0.743 using the minimum.
We believe that these experiments clearly establish that the success
of WRR's experiment was primarily due to the choices made in compiling
their lists and not to any genuine ELS phenomenon in Genesis. The data
for the above three experiments can be found at McKay's web site (1999b)."