Retrieving semantic information about specs and types

Erlang is dynamically typed, but it provides notations for types in the Erlang syntax. Although these do not affect the operation of the compiled code, it is generally a good practice to utilize them for type checking and other purposes, like doc generation. Our tool analyzes and stores "-type", "-opaque", "-export_type", "-spec" attributes and the types of record fields. These information can be conveniently accessed/used and comprehended via semantic queries.

Below is the brief description of available selectors in sq regarding type informations. For other available selectors, please visit our dedicated wiki page!

Initial selectors:

@spec → 	Spec
	The spec at the given position.
@type →	Type
	The type expression at the given position.

File selectors:

File.[spec, specs] → 	Spec
	Specs defined in the file.
File.typerefs → 	Type
	Types used in the file.
File.[typedef, typedefs, types, type] → 	Type
	Types defined in the file.

Function selectors:

Fun.spec → 	Spec
	Specification of the given function.
Fun.[returntype, returntypes, rettype] → 	Type
	Possible return types of the function according to the spec.
Fun.[arguments, argument, args, arg, parameters, parameter, params, param] → 	FunParam
	The arguments of the given function.

Function clause selectors:

FunClause.spec → 	Spec
	The specification of the function defining the given function clause.
FunClause.[returntype, returntypes, rettype] → 	Type
	Possible return types of the clause according to the spec.
FunClause.[arguments, argument, args, arg, parameters, parameter, params, param] → 	FunParam
	The arguments of the given function clause.

Spec selectors:

Spec.['fun', func, function]→ 	Fun
	The function that has this spec.
Spec.[file, deffile] → 	File
	The file in which the spec was defined.
Spec.[modref, refmod] → 	File
	The file to which the spec pertains.
Spec.[returntype, returntypes] → 	Type
	Possible return types of the spec.
Spec.[arguments, args, params, parameters, param, arg, argument] → 	Specparam
	Arguments of the spec.
Spec.[guardtype, guardtypes] → 	Type
	Types of the spec-guards.
Spec.returntext → 	string
	String representation of the specs return values.
Spec.guardtext → 	string
	String representation of the specs guards.
Spec.[name, funname] → 	atom
	Name of the function spec.
Spec.arity → 	int
	Arity of the spec.
Spec.text → 	string
	String representation of the spec.

Specparam selectors:

Specparam.type → 	Type
	The type of the spec-parameter.
Specparam.name → 	string
	Name of the spec-parameter.
Specparam.index → 	int
	Index of the spec-parameter.
Specparam.text → 	string
	String representation of the spec-parameter.

Type selectors:

Type.file → 	File
	The defining file of the type.
Type.[params, param, typeparams, typeparam, par, pars] → 	Type
	The parameters of the type.
Type.[subtype, subtypes, sub] → 	Type
	Subtypes of the given type.
Type.[specref, specrefs, refspec, refspecs] → 	Spec
	Specs using this type.
Type.[exported, remote_type] →	bool
	Returns whether the type is exported.
Type.name → 	atom
	The name of the type.
Type.[arity, number_of_parameters] → 	int
	The arity of the type.
Type.[isopaque, opaque, is_opaque] → 	bool
	Returns whether the type is opaque.
Type.[isbuiltin, is_builtin, is_built_in, builtin] → 	bool
	Returns whether the type is built-in.
Type.[refs, ref, references] → 	Type
	The references of the type.
Type.[paramsref, paramsrefs] → 	Type
	The references of the types arguments.

Other selectors:

FunParam.type → 	Type
	Possible types of the argument according to the spec.

RecordField.[type, types] → Type
	The type or type expression of the given record field.

Returned values for new internal types in sq:

As it can be seen in the above description, for the practical use of analyzed type informations, new internal types(spec, specparam, type) have been introduced for semantic queries. When a query isn't continued with other selectors to reach conventional types, it might not be obvious what the results would be. In case of our text-based and graphical interfaces, the format of the shown result for each of these types is as follows:

Spec -> string (textual representation of the -spec attribute = Spec.text)

SpecParam -> string (textual representation for the type expression of the spec-parameter = SpecParam.text)

Type -> string (Module:Name/Arity | Text)
Module -> atom (defining module of the type)
Name -> atom (name of the type = Type.name)
Arity -> int (number of parameters of the type = Type.arity)
Text -> string (textual representation of the type expression)

Examples:

ri:q("mods.specs").
a.erl
    -spec g(prp_user(X,Y))->'ok'.
...

ri:q("mods.specs.params").
-spec g(prp_user(X,Y))->'ok'.
    prp_user(X,Y)
...

ri:q("mods.types").
a.erl
    a:prp_user/2
    a:prp_user3/2
...

Our scriptable interface, along with the textual representations, also provides the unique entities of the stored type informations, which might be used to differentiate between results even when their formatted values are the same.

ris:q("mods.specs.params.type.params").
{rich,"a:prp_user/2\n    Key\n     Val\nb:prp_user/2\n    Key\n     Val\nc:prp_user/2\n    Key\n     Val\n",
      [{entity,{'$gn',typexp,151}},
       {entity,{'$gn',typexp,152}},
       {entity,{'$gn',typexp,296}},
       {entity,{'$gn',typexp,297}},
       {entity,{'$gn',typexp,439}},
       {entity,{'$gn',typexp,440}}]}

Detailed review of non-trivial selectors:

During the specification we have tried our best to keep things simple. This meant we would not differentiate between namedtypes(user-defined, built-in) and type expressions. In some cases this have resulted in rather complex specifications which can be seen below. Under normal circumstances, the different behaviors should be intuitively evident and would not cause any surprise.

File.typerefs:

This selector returns all types referenced in the file. It does not return type expressions, but it returns all other types(even built-in types or ones which have been defined in an other file) that have been referenced in "-spec", "-type", "-opaque" attributes or record definitions.

ri:q("mods.typerefs").
a.erl
    erlang:atom/0
    a:prp_user/2

Function.returntype(, Spec.returntype):

Returns types or type expressions present as the return-type of the function's specs. It does not split union types, but please note that specs can be overloaded so even functions with only one clause can have multiple values. (Single-atom type expressions are filtered.)

ri:q("mods.funs.returntypes").
a:g/2
    erlang:atom/0

Spec.refmod:

Specs can have module-identifiers which determine the module whose function is being specified. This module is returned by the selector unless the module-identifier was omitted, in which case the defining module is returned.

ri:q("mods.funs.spec.refmod").
-spec io:format(string(), list())->string().
    io

Spec.arguments:

Returns the type expression for the arguments of the specification. Argument-names are included.

ri:q("mods.funs.spec.params").
-spec io:format(Format :: string(), list())->string().
    Format :: string()
     list()

Spec.guardtypes:

Returns the type expressions of spec-guards.

ri:q("mods.funs.spec.guardtypes").
-spec io:format(Format :: string(), D)->string() when D :: list(string()).
     D :: list(string())

Specparam.type:

Returns the type or type expression of the given specparam. (Single-atom type expressions are filtered.)

ri:q("mods.funs.spec").           
io:format/2
    -spec io:format(string(), list)->string() when D :: list(string()).

ri:q("mods.funs.spec.param.type").
string()
    erlang:string/0

Specparam.name:

Returns the name of the specparam as a string. If the name was omitted, returns "undefined".

ri:q("mods.funs.spec").           
io:format/2
    -spec io:format(Format :: string(), list()) ->string().
ok
ri:q("mods.funs.spec.param.name").
Format :: string()
    name = "Format"
 list()
    name = "undefined"

Type.params:

Returns the type or type expression of the given type's parameters. (Single-atom type expressions are filtered.)

%%-type myt(Key, Val) :: myt(Key, ok).
ri:q("mods.types.params").           
a:myt/2
    Key
     Val
ok
ri:q("mods.types.subtypes.params").
 myt(Key, ok)
    Key

Type.subtypes:

If the given type is a user-defined type it returns the type's body as a type expression. When the given type is a type expression it returns the expression's subexpressions(1-level deep). (Single-atom type expressions are filtered.)

ri:q("mods.types.subtypes").
a:prp_user/2
     [{atom(), Key, Val}]
a:myt/2
     myt(Key, ok)
ok
ri:q("mods.types.subtypes.subtypes").
 [{atom(), Key, Val}]
    {atom(), Key, Val}
 myt(Key, ok)
    Key

Type.specrefs:

For user-defined and built-in types, returns the specs referencing this type.

ri:q("mods.types.specrefs").
b:prp_user/2
    -spec g(prp_user(X,Y))->'ok'.

Type.references:

Returns the type expressions using this user-defined or built-in type. If the given type is a type expression it tries to find the user-defined or built-in type referenced in the type expression and continues from there. Otherwise the result is empty.

ri:q("mods.types.refs").                  
b:prp_user/2
    prp_user(X,Y)
a:myt/2
     myt(list(), ok)
     myt(Key, ok)

Practical example:

Let's say we are new to a project with a large code-base and our task is to implement a new interface in which we have to do some operations on a very complex data structure. The source of this complex structure is remote, but we know there are multiple compatible data structures already implemented, although not sure about the specifics, so we would like to investigate and do this without manually going through several thousand lines of code.

We might start by running the following query:

"files.typerefs"

This gives us all used types in every file. After scrolling to the end of the hundred and something pages of results, we at least know types properly specified, but not much else. Maybe we should only look at types defined in "general" modules:

"mods[name ~ \"gen\"].types"

Unfortunately these seem to be either irrelevant or too general for us to use. We decide that we should look at types from other modules by filtering results using the "-spec" attributes (and let's assume, we are looking for a structure that has an "append" function implemented). But first we would like to know if "-spec" is used in our modules.

"mods.funs[not (.name in files.specs.name)][loc > 10]"

The query results are promising, it looks like most of the non-trivial functions have a specification. We can take a look at returned types of functions named like "append":

mods.funs[name ~ \"append\"].returntype

This query also gives us too many possible results, we are only interested in exported ones:

mods.funs[name ~ \"append\"].returntype[exported]

Turns out most of them were exported. However when we have looked at types from "gen" modules, we may have noticed a type("person") for a structure which is a part of what we have to append to the large structure.

mods.funs[name ~ \"append\"][.params.type.(subtype.(params)+)+[name = 'person']].returntype[exported]

This query only yields return-types of ~"append" functions that have a parameter whose type uses the "person" type. If we would like to see in which files these types have been used we could run the following:

mods.funs[name ~ \"append\"][.params.type.(subtype.(params)+)+[name = 'person']].returntype[exported].references.file

Limitations:

• For performance and readability reasons, type-expressions consisting of a single atom(e.g., bool, int, 'error') are filtered from results. This means, without re-parsing the textual representations, the results can not be used perform reliable type checking or comparison for functions. (We believe, including such simple expressions in results for an ordinarily large code-base would undermine the main goal of these features, which is providing an easy way for understanding type informations and their relation.)
• As of now, "@type" and "@spec" declarations are not analyzed.