Suffix tree based duplicate code analysis

In large program systems often occure duplicate codes, which is a computer programming term for a sequence of source code that occurs more than once. There are two ways in which two code sequences can be duplicates of each other: syntactically and functionally. This new feature can detect the syntactically similar duplicates, that only differ in atoms, constants (integer, float and string) and the name of variables. A lot of identical or very similar code fragments can result the application of copy and paste, for example.

Parameters

Parameters can be defined by a proplist. The possible properties are summed up in the table below. All the properties are optional.

Property	Description	Type	Default value	Example
files	specify files to search	[Module::atom() [[BR]] |Filepath::string() |RegExp::string() |File::string()]	all files from the database	{files,["/usr/local/lib/erlang/lib/mnesia-4.5/src/mnesia_log.erl", module]}
minlen	minimal length of a clone(length is in tokens)	integer()	10	{minlen,50}
minnum	minimal number of clones in one clone group	integer()	2	{minnum,5}
overlap	maximum length that duplicates can overlap each other (length is in tokens)	integer()	0	{overlap,1}
output	name of the file in which the results are saved	string()	-	{output,"result.txt"}
name	name of the result in the table	atom()	the atom 'temp' concatenated with the timestamp	{name,referl}

Interfaces

Web interface

See Duplicate code analysis on the web interface.

Console interface

Analyser functions

• search_duplicates/0: uses the default values of the properties.
• search_duplicates/1: takes a proplist as parameter described above.

Both interface function provide information about the progress of the process. The result is the list of clone groups. Every clones are defined by the path of the file and information about the start and end positions (line and column number). Results are saved. An example can be found at the bottom of the page.

Result query functions

Function	Parameters	Description	Example
stored_dupcode_results/0	-	queries all saved results (name and information about the parameters of the analysis)	ri:stored_dupcode_results()
save_dupcode_result/2	Name::atom() - name associated with the result, Filename::string() - name of the file	saves the given result in the file	ri:save_dupcode_result(temp20120527205412,"result.txt")
show_dupcode/1	Name::atom() - name associated with the result	lists the result's all clone groupes	ri:show_dupcode(temp20120527205412)
show_dupcode_group/2	Name::atom() - name associated with the result, GroupNumber::integer() - the required clone group's number	lists the given clone group of the result	ri:show_dupcode_group(temp20120527205412, 2)

Examples

Analysis functions

Allowed parameterizations:

ri:search_duplicates().

ri:search_duplicates([
        {files, [dup2,
                 "/home/user/dups/dup1.erl",
                 "/home/[0-9a-zA-Z/_.\-]+/src"]},
        {minlen, 50},
        {minnum, 3},
        {overlap, 1},
        {output, "result.txt"},
        {name, filtered}]).

Return value and the progress report (only if the analysis was not run yet):

(refactorerl@localhost)3> ri:search_duplicates([{files,[dup1,dup2]}]).

Initial clone detection started.
Initial clone detection finished.
Trimming clones started.
Trimming clones finished.
Filter clones finished.
Calculating positions started.
Calculating positions finished.
2 clone groups found.
Result saved to temp20120527234307...
run ri:show_dupcode(temp20120527234307) to see the result.
ok

(refactorerl@localhost)4> ri:search_duplicates([{files,[dup1,dup2]}]).
2 clone groups found.
Result saved to temp20120527234307...
run ri:show_dupcode(temp20120527234307) to see the result.
ok

Result query functions

(refactorerl@localhost)5> ri:stored_dupcode_results().
[[temp20120527234307,
  [{files,["/home/user/dups/dup1.erl",
           "/home/user/dups/dup2.erl"]},
   {minlen,10},
   {minnum,2},
   {overlap,0}]],
 [temp20120527235556,
  [{files,["/usr/local/lib/erlang/lib/mnesia-4.5/src/mnesia_schema.erl"]},
   {minlen,20},
   {minnum,3},
   {overlap,0}]]]
ok

(refactorerl@localhost)17> ri:show_dupcode(temp20120527234307).        
[{1,
  [[{filepath,"/home/user/dups/dup1.erl"},
    {startpos,{8,1}},
    {endpos,{18,11}}],
   [{filepath,"/home/user/dups/dup2.erl"},
    {startpos,{5,1}},
    {endpos,{15,11}}]]},
 {2,
  [[{filepath,"/home/user/dups/dup1.erl"},
    {startpos,{6,5}},
    {endpos,{6,62}}],
   [{filepath,"/home/user/dups/dup2.erl"},
    {startpos,{18,5}},
    {endpos,{18,58}}]]}]
Contains 2 clone groups...
run ri:show_dupcode_group(temp20120527234307, GroupNumber::integer()) to see one of the clone groups.
ok

(refactorerl@localhost)16> ri:show_dupcode_group(temp20120527234307,2).
[[{filepath,"/home/user/dups/dup1.erl"},
  {startpos,{6,5}},
  {endpos,{6,62}}],
 [{filepath,"/home/user/dups/dup2.erl"},
  {startpos,{18,5}},
  {endpos,{18,58}}]]
Contains 2 members.
ok