From: Eric S. Raymond Date: Wed, 10 Sep 2003 01:00:21 +0000 (+0000) Subject: Initial revision X-Git-Tag: 1.1~2 X-Git-Url: http://thyrsus.com/gitweb/?a=commitdiff_plain;h=691bc0e3cfea33f5e60239d7e27b5db7743d088e;p=comparator.git Initial revision --- diff --git a/comparator.xml b/comparator.xml new file mode 100644 index 0000000..ab892c8 --- /dev/null +++ b/comparator.xml @@ -0,0 +1,164 @@ + + + +comparator +1 +Sept 1 2003 + + +comparator +filterator +fast comparisons among large source trees + + + + + comparator + -c + -d dir + -o file + -s shredsize + -w + -x + path + + filterator + -d dir + -n + + + + +DESCRIPTION + +comparator is a program for quickly +finding common sections in two or more source-code trees. It's named +after a now-obsolete astronomical instrument called a blink comparator +that was used to find moving objects in starfield photos. +filterator is a postprocessor for +comparator reports that can apply +significance filtering. + +comparator works by first chopping +the specified trees into overlapping shreds (by default 3 lines long) +and computing the MD5 hash of each shred. The resulting list of +shreds is examined and all unique hashes are thrown out. Then +comparator generates a report listing all +cliques of lines with duplicate hashes, with overlapping ranges +merged. The output format can (not by coincidence) be stepped through +with the next-error function of Emacs. + +The program is capable of generating a hash list for a source +code tree which can be used in place of the tree itself. Thus, it is +possible to do comparisons with source trees without having access to +the actual source code, providing someone who has access is willing to +ship you a hash list. These hash lists are called ‘SCF +files’ and made with the extension .scf; +the name stands for Source Comparison Format. + +In its normal mode of operation, +comparator expects two or more command-line +arguments which are the root directories of the source-code trees to +be compared, and generates a report to standard output. Progress +messages are emitted to standard error. Some arguments may be the +names of SCF files. + +When given a single path argument which is a tree, the program +generates an SCF hash list to standard output. + +When the option is specified, the program +generates a SCF file from each tree specified on the command +line. The name of the resulting file is the argument name with +.scf appended. + +The directs output to a specified file. This +may be used with to override the normal +.scf convention, or simply as an alternative +to ouput redirection when generating a final report. + +The option changes current directory to the +specified tree before walking down each argument path to generate +hashes. This will be useful if you want to generate a report for +trees in a directory other than your current, but want the filenames +in the report to be relative. + +The option changes the shred size. Smaller +shred sizes are more sensitive to small code duplications, but produce +correspondingly noisier output. Larger ones will suppress both noise +and small similarities. + +The causes all whitespace in the file +(including blank lines) to be ignored for comparison purposes (line +numbers in the output report will nevertheless be correct). This is +recommended for comparing C code; among other things it means the +comparison won't be fooled by differences in indent style. + +The enables some debugging output. It +is for developers; if you need to understand it, read the code. + +comparator uses a grotty heuristic +for figuring out which files in a tree are eligible to be compared. +Files with .c and .h +extensions are always eligible. Files with .o or +~ extensions are always ignored. CVS, RCS, and +SCCS directories are ignored. Other files are checked to see if the +percentage of printable characters near the front of the file is above +90%. + +filterator is a postprocessor for the +SCF-B output of comparator that produces an +actual code listing. By default, it throws away ranges that appear to +be noise. The noise test is to throw away whitespace, C syntax, +#include lines and a few common macros and see if anything is left. +The suppresses noise filtering. The + acts as it does for +comparator. The option +sets the minimum size of overlap to be reported; setting this to a +slightly larger number than the comparator shred +size may be a useful way to filter out spurious matches. + +Occasionally the filterator output +looks as through the code has failed to properly merge two or more +overlapping line ranges, When this happens, the ranges have different +match sets; they may overlap in one tree, but not in others. A clue +to this is the number of matches reported for each range. + + + +AUTHOR +Eric S. Raymond esr@snark.thyrsus.com. See ESR's home +page at http://www.catb.org/~esr/ for +updates and other resources. + + +LIMITATIONS + +The heuristic for eligible files can be fooled, though this +is unlikely. + +The use of MD5 hashes means there is a very small probability of +false-positive matches between shreds — 1 in +18,446,744,073,709,551,616, to be precise. (You will probably be struck +by a meteor or spontaneously combust before this happens.) There +is no possibility of false negatives. + +You will get spurious results if you mix +.scf files with different shred sizes or +normalizations, or with settings for these that conflict with the +command-line options you have in effect for shredding source-code +trees. The program does some sanity checking but will not prevent you +from shooting yourself in the foot. + +The code for finding duplicate hashes uses a brute-force +approach that relies on being able to allocate core for the entire +hash list generated during processing. The results are almost +ridiculously fast on machines with enough core (the author has +observed speeds of up to two million lines of code compared per minute +on a 1.8GHz Intel box), but machines with small memories will thrash +their guts out as the VM subsystem tries to cope. + + +