312 lines
13 KiB
Markdown
312 lines
13 KiB
Markdown
# erpcgen test system
|
|
|
|
This file documents the parser and output test system for erpcgen.
|
|
|
|
This test system is built on py.test using its extensive plugin hooks.
|
|
|
|
## Setup
|
|
|
|
Python 3.6+ is required.
|
|
|
|
py.test(**Version 5.0.0-**) and pyYAML are required to run the tests. These can be installed via pip.
|
|
|
|
```sh
|
|
pip3 install pytest pyyaml
|
|
```
|
|
|
|
Make sure you have an erpcgen executable available. Either build it via the makefiles, or build
|
|
the VisualStudio project.
|
|
|
|
## Running
|
|
|
|
Run the tests by running the py.test executable. The tests can be run from either the
|
|
[erpc/erpcgen/](/erpcgen) or [erpc/erpcgen/test/](.) directories. The tests can be also run from root folder defining directory with tests as next parameter.
|
|
|
|
Run all tests:
|
|
|
|
```bash
|
|
$ pytest erpcgen/test
|
|
```
|
|
|
|
Run one test
|
|
|
|
```bash
|
|
$ pytest erpcgen/test/test_union_py.yml
|
|
```
|
|
|
|
Use the `--help` option of py.test to see all available command line options. Useful options are
|
|
`-s` to turn off output capturing and `-v` to list each test case by name as it runs. The `-k`
|
|
option lets you filter the test cases that are executed by a contains match on the name.
|
|
|
|
There is one erpcgen-specific command line option, `--erpcgen-verbosity` or `--ev`. This lets you
|
|
set the number of `-v` arguments that are passed to erpcgen calls. Specify an integer from 1 to 3,
|
|
such as `--ev=3`. Default verbosity is 0.
|
|
|
|
## Config
|
|
|
|
The tests use a config file. The standard config is [erpc/erpcgen/test/config.py](config.py). It tries to figure out the erpcgen executable location based on the OS. If you need to override this, you can create a `config_local.py` that defines the config variables listed below.
|
|
|
|
Config variables:
|
|
|
|
- `CC` = C compiler path, defaults to "gcc"
|
|
- `CXX` = C++ compiler path, defaults to "g++"
|
|
- `ERPCGEN` = absolute path to the erpcgen executable
|
|
- `RUN_KEEP_COUNT` = number of historical test runs to retain, not including the most recent run
|
|
|
|
The `ERPCGEN` and `CC` config variables can also be overridden with an environment variables of the
|
|
same name.
|
|
|
|
## Output
|
|
|
|
The files associated with test runs are stored in numbered directories under `erpc/erpcgen/test/runs/`.
|
|
By default, only the most recent 4 test runs are retained. See the `RUN_KEEP_COUNT` config variable.
|
|
A symlink called 'latest' is maintained to point at the most recent test run.
|
|
|
|
Test cases are stored in a hierarchy of directories under the test run directory:
|
|
|
|
`erpc/erpcgen/test/runs/<run#>/<yaml>/<spec>/[<params>]`
|
|
|
|
The "`<params>`" directory is added only if the test is parametrized. It will have a name based on
|
|
the unique set of parameter values for the test case.
|
|
|
|
Under the test case directory, the test IDL is stored in a `test.erpc` file. Generated files are
|
|
written by erpcgen into an `output/` subdirectory. Because the IDL file is named "test", the output
|
|
files names will by default also begin with "test", unless this is overridden through erpcgen
|
|
arguments or annotations.
|
|
|
|
If the erpcgen verbosity (see the `--erpcgen-verbosity` command line option above) is set to a
|
|
non-zero level, then the stdout output from erpcgen is captured in a `erpcgen.out` file in the test
|
|
case directory. This is useful for debugging issues.
|
|
|
|
For C language test cases, an `objects/` directory is created in the test case directory. It holds
|
|
compilation test inputs and outputs.
|
|
|
|
## Test specs
|
|
|
|
All .yml files in `erpc/erpcgen/test/` whose names begin with "test" are processed for test
|
|
specifications. Spec files can also be placed into subdirectories of `erpc/erpcgen/test/`, as long as
|
|
the subdirectory's name starts with "test".
|
|
|
|
Multiple test specs can be defined in each YAML file, separated by three dashes ("---"). (These are
|
|
multiple top-level YAML documents). Place the dashes before each test spec, not after. Each test
|
|
spec is a dictionary.
|
|
|
|
Test spec keys:
|
|
|
|
- `idl` = input erpcgen IDL
|
|
- `name` = short test name
|
|
- `desc` = description of the test case
|
|
- `lang` = output language (c or py); defaults to c
|
|
- `params` = parametrization, described below
|
|
- `args` = additional erpcgen command line arguments
|
|
- `jira` = JIRA issue key, for reference only
|
|
- (output filenames)
|
|
|
|
The only required key is `idl`.
|
|
|
|
Test specs are turned into one or more test cases via parametrization. If the `params` key is not
|
|
included in the spec, parametrization is disabled and only one test case will be produced. See
|
|
below for more about parametrization.
|
|
|
|
Each test case has its own directory under the main directory for the test run. When a test case is
|
|
run, the IDL is first written to a `test.erpc` file within the case directory. Then erpcgen is
|
|
executed with the output directory set to the `output/` subdirectory of the case directory. C code
|
|
is generated by default; this can be overridden with the `lang` spec key.
|
|
|
|
Filenames listed in the spec dictionary are relative to the `output/` directory of a test run. Under
|
|
the filenames are lists of test patterns that must all match against the file under which they are
|
|
listed for the test case to pass.
|
|
|
|
A basic test specification looks like this
|
|
|
|
```yml
|
|
---
|
|
name: quit_fn
|
|
idl: |
|
|
interface xyz {
|
|
oneway quit()
|
|
}
|
|
test.h:
|
|
- void quit(void)
|
|
```
|
|
|
|
This spec says to generate C output for the given IDL. It will match the string "`void quit(void)`"
|
|
against the `test.h` output file. If that string is found, the test will pass.
|
|
|
|
## Test patterns
|
|
|
|
Under each output filename in the test spec dictionary is a list of test patterns.
|
|
|
|
Test patterns must be listed in the order they will appear in the file. Each successive pattern will
|
|
start searching from the point where the last pattern matched. If you need to match multiple, out of
|
|
order, patterns then you can have the same output file listed more than once.
|
|
|
|
Patterns, by default, use simple literal string comparison.
|
|
|
|
Note that you must be aware of the YAML flow notation syntax. If you see YAML parse errors when you
|
|
run the tests, this is likely the cause. Patterns consisting of only one brace must be put in single
|
|
or double quotes. The same applies to patterns beginning with an open or close brace.
|
|
|
|
### Regular expressions
|
|
|
|
A pattern case be made a regular expression by turning it into a dict with a single `re` key. For
|
|
example:
|
|
|
|
```yml
|
|
- test.h:
|
|
- re: somevar[a-z]+
|
|
```
|
|
|
|
You must remember to escape all regex control characters with a backslash if you wish to match them
|
|
literally.
|
|
|
|
Literal patterns are actually converted to regexs internally, with appropriate escape of control
|
|
characters. Any space characters in literal patterns are replaced with "`\s*`", which will match
|
|
zero or more spaces. This is to make the patterns a little more resilient to minor changes in the
|
|
output formatting.
|
|
|
|
### Not patterns
|
|
|
|
Sometime you want to make sure that a certain string is _not_ present in the output. This can be
|
|
achieved by making a pattern a dict with a single `not` or `not_re` key. The former treats the
|
|
pattern as a literal match while the latter a regex.
|
|
|
|
Example:
|
|
|
|
```yml
|
|
- test/common.py:
|
|
- not: class Foo
|
|
- A = 1
|
|
```
|
|
|
|
Not patterns are searched for slightly differently than other patterns. Instead of searching to
|
|
the end of the file, not patterns are searched for within the range of the match positions of the
|
|
positive patterns before and after the group of not patterns, or the begin and end of the file if
|
|
there is not a positive pattern before or after the not pattern group. This limits the search range
|
|
of not patterns, and increases their effectiveness.
|
|
|
|
In the example above, the "`not: class Foo`" pattern is searched for from the start of the
|
|
`test/common.py` file to the position where the "`A = 1`" pattern successfully matched.
|
|
|
|
Consider this example:
|
|
|
|
```yml
|
|
- test/common.py:
|
|
- import erpc
|
|
- not: class Foo
|
|
- not_re: A = [0-9]+
|
|
- B = 1
|
|
```
|
|
|
|
Here, both not pattern searches range from the last matching character of the positive
|
|
"`import erpc`" pattern to the first matching character of the "`B = 1`" pattern. This example
|
|
also shows a `not_re` pattern that requires a regular expression to not match within the search
|
|
range.
|
|
|
|
## Parametrization
|
|
|
|
If the test spec dictionary has a `params` key, then the spec will be parametrized. The `params`
|
|
key must have a dictionary value with parameter names for keys. Under each of the parameter names
|
|
must be a list of values.
|
|
|
|
Here's an example valid `params` dict and IDL that uses the params:
|
|
|
|
```yml
|
|
params:
|
|
type:
|
|
- string
|
|
- binary
|
|
length:
|
|
- 1
|
|
- 8
|
|
idl: |
|
|
struct Foo {
|
|
{type}[{length}] my_array
|
|
}
|
|
```
|
|
|
|
A parametrized test generates test cases for all permutations of parameter values. In the above
|
|
examples, 4 test cases would be created for the 4 combinations of _type_ and _length_ param values.
|
|
Each param only has one value per test case. However, you can set an individual parameter value to a
|
|
list or dict (YAML flow notation works nicely for this). If you only have a single parameter, you
|
|
would get one test case for each value listed for that parameter.
|
|
|
|
The IDL and all test patterns will be formatted using parameter values. The Python `str.format()`
|
|
method is used. A substitution of the form "`{param_name}`" in the IDL or a test pattern will be
|
|
replaced with the parameter's value for that test case.
|
|
|
|
If a parameter value is a list or dict, you can substitute the elements by using index brackets:
|
|
"`{param_name[3]}`" or "`{param_name[dict_key]}`". Note that a dict key used in index brackets is
|
|
not placed in quotes. See the `str.format()` docs for full details of the format options.
|
|
|
|
Normally, `str.format()` will raise exceptions for open or close braces by themselves. Yet, the IDL
|
|
and C code frequently use braces. So, to simplify usage, the test spec strings are pre-processed to
|
|
convert isolated braces into the double braces required by `str.format()`. An open or close brace
|
|
must be followed by at least one whitespace character to be properly recognized and converted.
|
|
|
|
### If-then patterns
|
|
|
|
To make test specs even more flexible when using parametrization, simple if-then expressions are
|
|
allowed in the test patterns. This is done by making a test pattern a dictionary with `if`, `then`,
|
|
and optional `else` keys. The value of the `if` key is a Python Boolean expression, which is
|
|
formatted with `str.format()` (like all test pattern strings) prior to evaluation. The namespace
|
|
of the if expression has all parameter names set to the values for the test case.
|
|
|
|
The value of the `then` key is a list of test patterns which will be used if the if predicate
|
|
evaluates to true. If an `else` key is present, its value is a list of test patterns that will be
|
|
used only if the if predicate evaluates to false.
|
|
|
|
Example if-then pattern:
|
|
|
|
```yml
|
|
test/common.py:
|
|
- if: type=="string" # type is a param
|
|
then:
|
|
- start_read_list
|
|
- re: read_[a-z]+ # regex test pattern
|
|
- end_read_list
|
|
else:
|
|
- read_string
|
|
```
|
|
|
|
If-then patterns can be nested, and include regex and not patterns as shown in the example above.
|
|
However, "else if" expressions are not supported. If you need an "else if", you can use multiple
|
|
if-then patterns.
|
|
|
|
## Compilation testing
|
|
|
|
The erpcgen output from each of the test cases is run through the C/C++ compiler or Python. This is
|
|
done only to test syntax. The code is not linked into an executable.
|
|
|
|
C language tests have an `objects/` directory created in the test case directory. Source files
|
|
named `main_c.c` and `main_cxx.cpp` are written into this directory, and are included in the
|
|
compile. These files test that the common generated header file can successfully be included by both
|
|
the C and C++ compilers. The `objects/` directory also contains .o files written by the compiler.
|
|
|
|
For Python test cases, the generated package is loaded and compiled. Each of the modules within the
|
|
package are also loaded and compiled explicitly. Loading is done directly within the Python
|
|
executable running py.test.
|
|
|
|
The `CC` config variable is used for the path to the C/C++ compiler. The default value is simply
|
|
"gcc". You can override this with a `config_local.py` file, or an environment variable.
|
|
|
|
Compilation testing is not currently performed on Windows systems.
|
|
|
|
## Tips for writing test specs
|
|
|
|
1. Keep each test spec simple and focused.
|
|
2. Test only one feature at a time.
|
|
3. Test patterns should focus on the key identifiers and structural elements; don't necessarily make
|
|
a pattern match an entire output line. This will help keep the tests more flexible and resilient
|
|
to generator code changes, yet still verify that the important parts of the output are present.
|
|
4. Use the fact that patterns must match in order with the file to improve the accuracy of your
|
|
tests, by adding "anchor" patterns before and after your patterns to ensure they match within a
|
|
certain area of the output.
|
|
5. Use parametrization to produce many test cases with few test specs.
|
|
6. If-then patterns can expand the number of cases you can generate with a single spec.
|
|
7. Not patterns are useful in pairs or sets of cases that test a feature both enabled and disabled,
|
|
so you can verify that the output does and does not include appropriate code for the feature.
|
|
8. Complex tests can be written that test multiple features, but these should be written only after
|
|
the individual features are thoroughly covered.
|
|
9. Use YAML literal block scalars for the IDL so whitespace is not compressed in `test.erpc` files.
|