Changes between Initial Version and Version 1 of SemanticQuery/Highlights

Timestamp:

Sep 17, 2014, 4:45:50 PM (10 years ago)

Author:

manualwiki

Comment:

--

SemanticQuery/Highlights

@@ +1 @@
+= Retrieving semantic information about specs and types =
+
+Erlang is dynamically typed, but it provides notations for types in the Erlang syntax. Although these do not affect the operation of the compiled code, it is generally a good practice to utilize them for type checking and other purposes, like doc generation. Our tool analyzes and stores "-type", "-opaque", "-export_type", "-spec" attributes and the types of record fields. These information can be conveniently accessed/used and comprehended via semantic queries.
+
+Below is the brief description of available selectors in sq regarding type informations. For other available selectors, please visit our dedicated wiki page!
+
+=== Initial selectors: ===
+{{{
+@spec →         Spec
+        The spec at the given position.
+@type → Type
+        The type expression at the given position.
+}}}
+
+=== File selectors: ===
+{{{
+File.[spec, specs] →    Spec
+        Specs defined in the file.
+File.typerefs →         Type
+        Types used in the file.
+File.[typedef, typedefs, types, type] →         Type
+        Types defined in the file.
+}}}
+
+=== Function selectors: ===
+{{{
+Fun.spec →      Spec
+        Specification of the given function.
+Fun.[returntype, returntypes, rettype] →        Type
+        Possible return types of the function according to the spec.
+Fun.[arguments, argument, args, arg, parameters, parameter, params, param] →    FunParam
+        The arguments of the given function.
+}}}
+
+=== Function clause selectors: ===
+{{{
+FunClause.spec →        Spec
+        The specification of the function defining the given function clause.
+FunClause.[returntype, returntypes, rettype] →  Type
+        Possible return types of the clause according to the spec.
+FunClause.[arguments, argument, args, arg, parameters, parameter, params, param] →      FunParam
+        The arguments of the given function clause.
+}}}
+
+=== Spec selectors: ===
+{{{
+Spec.['fun', func, function]→   Fun
+        The function that has this spec.
+Spec.[file, deffile] →  File
+        The file in which the spec was defined.
+Spec.[modref, refmod] →         File
+        The file to which the spec pertains.
+Spec.[returntype, returntypes] →        Type
+        Possible return types of the spec.
+Spec.[arguments, args, params, parameters, param, arg, argument] →      Specparam
+        Arguments of the spec.
+Spec.[guardtype, guardtypes] →  Type
+        Types of the spec-guards.
+Spec.returntext →       string
+        String representation of the specs return values.
+Spec.guardtext →        string
+        String representation of the specs guards.
+Spec.[name, funname] →  atom
+        Name of the function spec.
+Spec.arity →    int
+        Arity of the spec.
+Spec.text →     string
+        String representation of the spec.
+}}}
+
+=== Specparam selectors: ===
+{{{
+Specparam.type →        Type
+        The type of the spec-parameter.
+Specparam.name →        string
+        Name of the spec-parameter.
+Specparam.index →       int
+        Index of the spec-parameter.
+Specparam.text →        string
+        String representation of the spec-parameter.
+}}}
+
+=== Type selectors: ===
+{{{
+Type.file →     File
+        The defining file of the type.
+Type.[params, param, typeparams, typeparam, par, pars] →        Type
+        The parameters of the type.
+Type.[subtype, subtypes, sub] →         Type
+        Subtypes of the given type.
+Type.[specref, specrefs, refspec, refspecs] →   Spec
+        Specs using this type.
+Type.[exported, remote_type] →  bool
+        Returns whether the type is exported.
+Type.name →     atom
+        The name of the type.
+Type.[arity, number_of_parameters] →    int
+        The arity of the type.
+Type.[isopaque, opaque, is_opaque] →    bool
+        Returns whether the type is opaque.
+Type.[isbuiltin, is_builtin, is_built_in, builtin] →    bool
+        Returns whether the type is built-in.
+Type.[refs, ref, references] →  Type
+        The references of the type.
+Type.[paramsref, paramsrefs] →  Type
+        The references of the types arguments.
+}}}
+
+=== Other selectors: ===
+{{{
+FunParam.type →         Type
+        Possible types of the argument according to the spec.
+
+RecordField.[type, types] → Type
+        The type or type expression of the given record field.
+}}}
+
+== Returned values for new internal types in sq: ==
+As it can be seen in the above description, for the practical use of analyzed type informations, new internal types(spec, specparam, type) have been introduced for semantic queries. When a query isn't continued with other selectors to reach conventional types, it might not be obvious what the results would be. In case of our text-based and graphical interfaces, the format of the shown result for each of these types is as follows:
+
+{{{
+Spec -> string (textual representation of the -spec attribute = Spec.text)
+
+SpecParam -> string (textual representation for the type expression of the spec-parameter = SpecParam.text)
+
+Type -> string (Module:Name/Arity | Text)
+Module -> atom (defining module of the type)
+Name -> atom (name of the type = Type.name)
+Arity -> int (number of parameters of the type = Type.arity)
+Text -> string (textual representation of the type expression)
+}}}
+
+Examples:
+{{{
+ri:q("mods.specs").
+a.erl
+    -spec g(prp_user(X,Y))->'ok'.
+...
+
+ri:q("mods.specs.params").
+-spec g(prp_user(X,Y))->'ok'.
+    prp_user(X,Y)
+...
+
+ri:q("mods.types").
+a.erl
+    a:prp_user/2
+    a:prp_user3/2
+...
+}}}
+
+Our scriptable interface, along with the textual representations, also provides the unique entities of the stored type informations, which might be used to differentiate between results even when their formatted values are the same.
+
+{{{
+ris:q("mods.specs.params.type.params").
+{rich,"a:prp_user/2\n    Key\n     Val\nb:prp_user/2\n    Key\n     Val\nc:prp_user/2\n    Key\n     Val\n",
+      [{entity,{'$gn',typexp,151}},
+       {entity,{'$gn',typexp,152}},
+       {entity,{'$gn',typexp,296}},
+       {entity,{'$gn',typexp,297}},
+       {entity,{'$gn',typexp,439}},
+       {entity,{'$gn',typexp,440}}]}
+}}}
+
+== Detailed review of non-trivial selectors: ==
+
+During the specification we have tried our best to keep things simple. This meant we would not differentiate between namedtypes(user-defined, built-in) and type expressions. In some cases this have resulted in rather complex specifications which can be seen below. Under normal circumstances, the different behaviors should be intuitively evident and would not cause any surprise.
+
+=== File.typerefs: ===
+This selector returns all types referenced in the file. It does not return type expressions, but it returns all other types(even built-in types or ones which have been defined in an other file) that have been referenced in "-spec", "-type", "-opaque" attributes or record definitions.
+{{{
+ri:q("mods.typerefs").
+a.erl
+    erlang:atom/0
+    a:prp_user/2
+}}}
+
+=== Function.returntype(, Spec.returntype): ===
+Returns types or type expressions present as the return-type of the function's specs. It does not split union types, but please note that specs can be overloaded so even functions with only one clause can have multiple values. (Single-atom type expressions are filtered.)
+{{{
+ri:q("mods.funs.returntypes").
+a:g/2
+    erlang:atom/0
+}}}
+
+=== Spec.refmod: ===
+Specs can have module-identifiers which determine the module whose function is being specified. This module is returned by the selector unless the module-identifier was omitted, in which case the defining module is returned.
+{{{
+ri:q("mods.funs.spec.refmod").
+-spec io:format(string(), list())->string().
+    io
+}}}
+
+=== Spec.arguments: ===
+Returns the type expression for the arguments of the specification. Argument-names are included.
+{{{
+ri:q("mods.funs.spec.params").
+-spec io:format(Format :: string(), list())->string().
+    Format :: string()
+     list()
+}}}
+
+=== Spec.guardtypes: ===
+Returns the type expressions of spec-guards.
+{{{
+ri:q("mods.funs.spec.guardtypes").
+-spec io:format(Format :: string(), D)->string() when D :: list(string()).
+     D :: list(string())
+}}}
+
+=== Specparam.type: ===
+Returns the type or type expression of the given specparam. (Single-atom type expressions are filtered.)
+{{{
+ri:q("mods.funs.spec").           
+io:format/2
+    -spec io:format(string(), list)->string() when D :: list(string()).
+
+ri:q("mods.funs.spec.param.type").
+string()
+    erlang:string/0
+}}}
+
+=== Specparam.name: ===
+Returns the name of the specparam as a string. If the name was omitted, returns "undefined".
+{{{
+ri:q("mods.funs.spec").           
+io:format/2
+    -spec io:format(Format :: string(), list()) ->string().
+ok
+ri:q("mods.funs.spec.param.name").
+Format :: string()
+    name = "Format"
+ list()
+    name = "undefined"
+}}}
+
+=== Type.params: ===
+Returns the type or type expression of the given type's parameters. (Single-atom type expressions are filtered.)
+{{{
+%%-type myt(Key, Val) :: myt(Key, ok).
+ri:q("mods.types.params").           
+a:myt/2
+    Key
+     Val
+ok
+ri:q("mods.types.subtypes.params").
+ myt(Key, ok)
+    Key
+}}}
+
+=== Type.subtypes: ===
+If the given type is a user-defined type it returns the type's body as a type expression. When the given type is a type expression it returns the expression's subexpressions(1-level deep). (Single-atom type expressions are filtered.)
+{{{
+ri:q("mods.types.subtypes").
+a:prp_user/2
+     [{atom(), Key, Val}]
+a:myt/2
+     myt(Key, ok)
+ok
+ri:q("mods.types.subtypes.subtypes").
+ [{atom(), Key, Val}]
+    {atom(), Key, Val}
+ myt(Key, ok)
+    Key
+}}}
+
+=== Type.specrefs: ===
+For user-defined and built-in types, returns the specs referencing this type.
+{{{
+ri:q("mods.types.specrefs").
+b:prp_user/2
+    -spec g(prp_user(X,Y))->'ok'.
+}}}
+
+=== Type.references: ===
+Returns the type expressions using this user-defined or built-in type. If the given type is a type expression it tries to find the user-defined or built-in type referenced in the type expression and continues from there. Otherwise the result is empty.
+{{{
+ri:q("mods.types.refs").                  
+b:prp_user/2
+    prp_user(X,Y)
+a:myt/2
+     myt(list(), ok)
+     myt(Key, ok)
+}}}
+
+== Practical example: ==
+Let's say we are new to a project with a large code-base and our task is to implement a new interface in which we have to do some operations on a very complex data structure. The source of this complex structure is remote, but we know there are multiple compatible data structures already implemented, although not sure about the specifics, so we would like to investigate and do this without manually going through several thousand lines of code.
+
+We might start by running the following query:
+{{{
+"files.typerefs"
+}}}
+This gives us all used types in every file. After scrolling to the end of the hundred and something pages of results, we at least know types properly specified, but not much else.
+Maybe we should only look at types defined in "general" modules:
+{{{
+"mods[name ~ \"gen\"].types"
+}}}
+Unfortunately these seem to be either irrelevant or too general for us to use. We decide that we should look at types from other modules by filtering results using the "-spec" attributes (and let's assume, we are looking for a structure that has an "append" function implemented).
+But first we would like to know if "-spec" is used in our modules.
+{{{
+"mods.funs[not (.name in files.specs.name)][loc > 10]"
+}}}
+The query results are promising, it looks like most of the non-trivial functions have a specification.
+We can take a look at returned types of functions named like "append":
+{{{
+mods.funs[name ~ \"append\"].returntype
+}}}
+This query also gives us too many possible results, we are only interested in exported ones:
+{{{
+mods.funs[name ~ \"append\"].returntype[exported]
+}}}
+Turns out most of them were exported. However when we have looked at types from "gen" modules, we may have noticed a type("person") for a structure which is a part of what we have to append to the large structure.
+{{{
+mods.funs[name ~ \"append\"][.params.type.(subtype.(params)+)+[name = 'person']].returntype[exported]
+}}}
+This query only yields return-types of ~"append" functions that have a parameter whose type uses the "person" type.
+If we would like to see in which files these types have been used we could run the following:
+{{{
+mods.funs[name ~ \"append\"][.params.type.(subtype.(params)+)+[name = 'person']].returntype[exported].references.file
+}}}
+
+== Limitations: ==
+ - For performance and readability reasons, type-expressions consisting of a single atom(e.g., bool, int, 'error') are filtered from results. This means, without re-parsing the textual representations, the results can not be used perform reliable type checking or comparison for functions. (We believe, including such simple expressions in results for an ordinarily large code-base would undermine the main goal of these features, which is providing an easy way for understanding type informations and their relation.)
+ - As of now, "@type" and "@spec" declarations are not analyzed.