From: Eric S. Raymond <esr@thyrsus.com>
Date: Mon, 15 Sep 2003 16:53:14 +0000 (+0000)
Subject: We now have two hash methods available.
X-Git-Tag: 1.6~3
X-Git-Url: http://thyrsus.com/gitweb/?a=commitdiff_plain;h=2571de95d002f57af237110ca7607f55dc0aafba;p=comparator.git

We now have two hash methods available.
---

diff --git a/Makefile b/Makefile
index 171c373..4b3ebb8 100644
--- a/Makefile
+++ b/Makefile
@@ -3,7 +3,7 @@
 # By building with -DLARGEFILES you can go to all 32-byte offsets.
 # This increases working-set size by 20% but handles > 65336 lines per file.
 
-VERS=1.6
+VERS=2.0
 
 CODE    = shredtree.c shred.h report.c hash.c main.c md5.c md5.h \
 		hash.h hashtab.h \
@@ -28,7 +28,7 @@ report.o: report.c shred.h hash.h
 comparator: main.o hash.o shredtree.o md5.o report.o
 	$(CC) $(CFLAGS) main.o hash.o shredtree.o md5.o report.o $(LDFLAGS) -o comparator
 
-hash.c: hashgen.py
+hashtab.h: hashgen.py
 	python hashgen.py >hashtab.h
 
 clean:
diff --git a/comparator.xml b/comparator.xml
index eddcae5..d32258e 100644
--- a/comparator.xml
+++ b/comparator.xml
@@ -48,7 +48,7 @@ can apply significance filtering.</para>
 
 <para><application>comparator</application> works by first chopping
 the specified trees into overlapping shreds (by default 3 lines long)
-and computing the MD5 hash of each shred.  The resulting list of
+and computing a hash of each shred.  The resulting list of
 shreds is examined and all unique hashes are thrown out.  Then
 <application>comparator</application> generates a report listing all
 cliques of lines with duplicate hashes, with overlapping ranges
@@ -164,9 +164,10 @@ to this is the number of matches reported for each range.</para>
 
 </refsect1>
 
-<refsect1 id='author'><title>AUTHOR</title>
-<para>Eric S. Raymond <email>esr@snark.thyrsus.com</email>.  See ESR's home
-page at <ulink
+<refsect1 id='author'><title>AUTHOR</title> <para>Eric S. Raymond
+<email>esr@snark.thyrsus.com</email>.  Ronald Rivest suggested the
+custom RXOR hash function used in versions 2.0 and up.  See ESR's home page
+at <ulink
 url='http://www.catb.org/~esr/'>http://www.catb.org/~esr/</ulink> for
 updates and other resources.</para>
 </refsect1>
@@ -185,11 +186,13 @@ into the territory of ideas and tend to produce false positives.</para>
 <para>The heuristic for eligible files can be fooled, though this
 is unlikely.</para>
 
-<para>The use of MD5 hashes means there is a very small probability of
-false-positive matches between shreds &mdash; 1 in
-18,446,744,073,709,551,616, to be precise. (You will probably be struck
-by a meteor or spontaneously combust before this happens.)  There
-is no possibility of false negatives.</para>
+<para>The use of hashes means there is a very small probability of
+false-positive matches between shreds; to be precise, with the default
+RXOR hash you can expect the first false positive after accumulating
+6,074,000,999 shreds.  For even very large forests, such as
+collections of Linux and Unix source trees running into the tens of
+millions of lines, you are very unlikely to ever see this.  There is
+no possibility of false negatives.</para>
 
 <para>This program handles a maximum of 65536 lines per file.  A full
 integer line-number range would increase working-set size by
diff --git a/hash.c b/hash.c
index 674eac4..4c8a805 100644
--- a/hash.c
+++ b/hash.c
@@ -1,7 +1,7 @@
 /* hash.c -- hash and digest computation for comparator */
 #include "hash.h"
 
-#ifdef INTEGRAL_HASH
+#ifndef USE_MD5
 /******************************************************************************
 
 According to the Birthday Theorem, successive random choices from a
diff --git a/hash.h b/hash.h
index f20a6a7..5c1c331 100644
--- a/hash.h
+++ b/hash.h
@@ -1,9 +1,11 @@
 /* interface file for comparator hash function */
 
-#ifdef INTEGRAL_HASH
+#ifndef FORCE_MD5
 typedef unsigned long long	hashval_t;
+#define HASHMETHOD	"RXOR"
 #else
 typedef unsigned char	hashval_t[16];
+#define HASHMETHOD	"MD5"
 #endif
 
 #define hash_compare(s, t)	memcmp(&(s), &(t), sizeof(hashval_t))
diff --git a/scf-standard.xml b/scf-standard.xml
index d8a18c6..8beb45c 100644
--- a/scf-standard.xml
+++ b/scf-standard.xml
@@ -19,6 +19,14 @@
 
 <revhistory> 
    <revision>
+      <revnumber>2.0</revnumber>
+      <date>2003-12-15</date>
+      <authorinitials>esr</authorinitials>
+       <revremark>
+	 New custom hash function; binary format changes accordingly.
+       </revremark>
+  </revision>
+  <revision>
       <revnumber>1.3</revnumber>
       <date>2003-12-07</date>
       <authorinitials>esr</authorinitials>
@@ -249,7 +257,8 @@ may include the following:</para>
 
 <varlistentry>
 <term><emphasis>Hash-Method</emphasis>: (optional)</term>
-<listitem><para>The hashing method.  Defaults to MD5</para></listitem>
+<listitem><para>The hashing method.  Defaults to RXOR for the custom
+Rivest XOR hash; MD5 is also a legal value.</para></listitem>
 </varlistentry>
 
 <varlistentry>
@@ -271,8 +280,8 @@ records.  The length of the data section is drtivable from the data
 lengths.</para>
 
 <para>A shred record consists of ushorts (start and end line numbers)
-followed by a fixed-length block of hash data.  For the default MD5
-hash the data is 16 bytes.</para>
+followed by a fixed-length block of hash data.  For the default RXOR
+hash the data is 8 bytes; for MD5, it is 16.</para>
 
 <para>Shreds should be ordered by start line number.  Empty files should be
 skipped, and nonempty files should always have at least one shred.