# erpcgen test system

This file documents the parser and output test system for erpcgen.

This test system is built on py.test using its extensive plugin hooks.

## Setup

Python 3.6+ is required.

py.test(**Version 5.0.0-**) and pyYAML are required to run the tests. These can be installed via pip.

```sh
pip3 install pytest pyyaml
```

Make sure you have an erpcgen executable available. Either build it via the makefiles, or build
the VisualStudio project.

## Running

Run the tests by running the py.test executable.  The tests can be run from either the
[erpc/erpcgen/](/erpcgen) or [erpc/erpcgen/test/](.) directories. The tests can be also run from root folder defining directory with tests as next parameter.

Run all tests:

```bash
$ pytest erpcgen/test
```

Run one test

```bash
$ pytest erpcgen/test/test_union_py.yml
```

Use the `--help` option of py.test to see all available command line options. Useful options are
`-s` to turn off output capturing and `-v` to list each test case by name as it runs. The `-k`
option lets you filter the test cases that are executed by a contains match on the name.

There is one erpcgen-specific command line option, `--erpcgen-verbosity` or `--ev`. This lets you
set the number of `-v` arguments that are passed to erpcgen calls. Specify an integer from 1 to 3,
such as `--ev=3`. Default verbosity is 0.

## Config

The tests use a config file. The standard config is [erpc/erpcgen/test/config.py](config.py). It tries to figure out the erpcgen executable location based on the OS. If you need to override this, you can create a `config_local.py` that defines the config variables listed below.

Config variables:

- `CC` = C compiler path, defaults to "gcc"
- `CXX` = C++ compiler path, defaults to "g++"
- `ERPCGEN` = absolute path to the erpcgen executable
- `RUN_KEEP_COUNT` = number of historical test runs to retain, not including the most recent run

The `ERPCGEN` and `CC` config variables can also be overridden with an environment variables of the
same name.

## Output

The files associated with test runs are stored in numbered directories under `erpc/erpcgen/test/runs/`.
By default, only the most recent 4 test runs are retained. See the `RUN_KEEP_COUNT` config variable.
A symlink called 'latest' is maintained to point at the most recent test run.

Test cases are stored in a hierarchy of directories under the test run directory:

`erpc/erpcgen/test/runs/<run#>/<yaml>/<spec>/[<params>]`

The "`<params>`" directory is added only if the test is parametrized. It will have a name based on
the unique set of parameter values for the test case.

Under the test case directory, the test IDL is stored in a `test.erpc` file. Generated files are
written by erpcgen into an `output/` subdirectory. Because the IDL file is named "test", the output
files names will by default also begin with "test", unless this is overridden through erpcgen
arguments or annotations.

If the erpcgen verbosity (see the `--erpcgen-verbosity` command line option above) is set to a
non-zero level, then the stdout output from erpcgen is captured in a `erpcgen.out` file in the test
case directory. This is useful for debugging issues.

For C language test cases, an `objects/` directory is created in the test case directory. It holds
compilation test inputs and outputs.

## Test specs

All .yml files in `erpc/erpcgen/test/` whose names begin with "test" are processed for test
specifications. Spec files can also be placed into subdirectories of `erpc/erpcgen/test/`, as long as
the subdirectory's name starts with "test".

Multiple test specs can be defined in each YAML file, separated by three dashes ("---"). (These are
multiple top-level YAML documents). Place the dashes before each test spec, not after. Each test
spec is a dictionary.

Test spec keys:

- `idl` = input erpcgen IDL
- `name` = short test name
- `desc` = description of the test case
- `lang` = output language (c or py); defaults to c
- `params` = parametrization, described below
- `args` = additional erpcgen command line arguments
- `jira` = JIRA issue key, for reference only
- (output filenames)

The only required key is `idl`.

Test specs are turned into one or more test cases via parametrization. If the `params` key is not
included in the spec, parametrization is disabled and only one test case will be produced. See
below for more about parametrization.

Each test case has its own directory under the main directory for the test run. When a test case is
run, the IDL is first written to a `test.erpc` file within the case directory. Then erpcgen is
executed with the output directory set to the `output/` subdirectory of the case directory. C code
is generated by default; this can be overridden with the `lang` spec key.

Filenames listed in the spec dictionary are relative to the `output/` directory of a test run. Under
the filenames are lists of test patterns that must all match against the file under which they are
listed for the test case to pass.

A basic test specification looks like this

```yml
---
name: quit_fn
idl: |
    interface xyz {
    oneway quit()
    }
test.h:
    - void quit(void)
```

This spec says to generate C output for the given IDL. It will match the string "`void quit(void)`"
against the `test.h` output file. If that string is found, the test will pass.

## Test patterns

Under each output filename in the test spec dictionary is a list of test patterns.

Test patterns must be listed in the order they will appear in the file. Each successive pattern will
start searching from the point where the last pattern matched. If you need to match multiple, out of
order, patterns then you can have the same output file listed more than once.

Patterns, by default, use simple literal string comparison.

Note that you must be aware of the YAML flow notation syntax. If you see YAML parse errors when you
run the tests, this is likely the cause. Patterns consisting of only one brace must be put in single
or double quotes. The same applies to patterns beginning with an open or close brace.

### Regular expressions

A pattern case be made a regular expression by turning it into a dict with a single `re` key. For
example:

```yml
- test.h:
    - re: somevar[a-z]+
```

You must remember to escape all regex control characters with a backslash if you wish to match them
literally.

Literal patterns are actually converted to regexs internally, with appropriate escape of control
characters. Any space characters in literal patterns are replaced with "`\s*`", which will match
zero or more spaces. This is to make the patterns a little more resilient to minor changes in the
output formatting.

### Not patterns

Sometime you want to make sure that a certain string is _not_ present in the output. This can be
achieved by making a pattern a dict with a single `not` or `not_re` key. The former treats the
pattern as a literal match while the latter a regex.

Example:

```yml
- test/common.py:
    - not: class Foo
    - A = 1
```

Not patterns are searched for slightly differently than other patterns. Instead of searching to
the end of the file, not patterns are searched for within the range of the match positions of the
positive patterns before and after the group of not patterns, or the begin and end of the file if
there is not a positive pattern before or after the not pattern group. This limits the search range
of not patterns, and increases their effectiveness.

In the example above, the "`not: class Foo`" pattern is searched for from the start of the
`test/common.py` file to the position where the "`A = 1`" pattern successfully matched.

Consider this example:

```yml
- test/common.py:
    - import erpc
    - not: class Foo
    - not_re: A = [0-9]+
    - B = 1
```

Here, both not pattern searches range from the last matching character of the positive
"`import erpc`" pattern to the first matching character of the "`B = 1`" pattern. This example
also shows a `not_re` pattern that requires a regular expression to not match within the search
range.

## Parametrization

If the test spec dictionary has a `params` key, then the spec will be parametrized. The `params`
key must have a dictionary value with parameter names for keys. Under each of the parameter names
must be a list of values.

Here's an example valid `params` dict and IDL that uses the params:

```yml
    params:
        type:
            - string
            - binary
        length:
            - 1
            - 8
    idl: |
        struct Foo {
            {type}[{length}] my_array
        }
```

A parametrized test generates test cases for all permutations of parameter values. In the above
examples, 4 test cases would be created for the 4 combinations of _type_ and _length_ param values.
Each param only has one value per test case. However, you can set an individual parameter value to a
list or dict (YAML flow notation works nicely for this). If you only have a single parameter, you
would get one test case for each value listed for that parameter.

The IDL and all test patterns will be formatted using parameter values. The Python `str.format()`
method is used. A substitution of the form "`{param_name}`" in the IDL or a test pattern will be
replaced with the parameter's value for that test case.

If a parameter value is a list or dict, you can substitute the elements by using index brackets:
"`{param_name[3]}`" or "`{param_name[dict_key]}`". Note that a dict key used in index brackets is
not placed in quotes. See the `str.format()` docs for full details of the format options.

Normally, `str.format()` will raise exceptions for open or close braces by themselves. Yet, the IDL
and C code frequently use braces. So, to simplify usage, the test spec strings are pre-processed to
convert isolated braces into the double braces required by `str.format()`. An open or close brace
must be followed by at least one whitespace character to be properly recognized and converted.

### If-then patterns

To make test specs even more flexible when using parametrization, simple if-then expressions are
allowed in the test patterns. This is done by making a test pattern a dictionary with `if`, `then`,
and optional `else` keys. The value of the `if` key is a Python Boolean expression, which is
formatted with `str.format()` (like all test pattern strings) prior to evaluation. The namespace
of the if expression has all parameter names set to the values for the test case.

The value of the `then` key is a list of test patterns which will be used if the if predicate
evaluates to true. If an `else` key is present, its value is a list of test patterns that will be
used only if the if predicate evaluates to false.

Example if-then pattern:

```yml
test/common.py:
    - if: type=="string"    # type is a param
    then:
        - start_read_list
        - re: read_[a-z]+     # regex test pattern
        - end_read_list
    else:
        - read_string
```

If-then patterns can be nested, and include regex and not patterns as shown in the example above.
However, "else if" expressions are not supported. If you need an "else if", you can use multiple
if-then patterns.

## Compilation testing

The erpcgen output from each of the test cases is run through the C/C++ compiler or Python. This is
done only to test syntax. The code is not linked into an executable.

C language tests have an `objects/` directory created in the test case directory. Source files
named `main_c.c` and `main_cxx.cpp` are written into this directory, and are included in the
compile. These files test that the common generated header file can successfully be included by both
the C and C++ compilers. The `objects/` directory also contains .o files written by the compiler.

For Python test cases, the generated package is loaded and compiled. Each of the modules within the
package are also loaded and compiled explicitly. Loading is done directly within the Python
executable running py.test.

The `CC` config variable is used for the path to the C/C++ compiler. The default value is simply
"gcc". You can override this with a `config_local.py` file, or an environment variable.

Compilation testing is not currently performed on Windows systems.

## Tips for writing test specs

1. Keep each test spec simple and focused.
2. Test only one feature at a time.
3. Test patterns should focus on the key identifiers and structural elements; don't necessarily make
   a pattern match an entire output line. This will help keep the tests more flexible and resilient
   to generator code changes, yet still verify that the important parts of the output are present.
4. Use the fact that patterns must match in order with the file to improve the accuracy of your
   tests, by adding "anchor" patterns before and after your patterns to ensure they match within a
   certain area of the output.
5. Use parametrization to produce many test cases with few test specs.
6. If-then patterns can expand the number of cases you can generate with a single spec.
7. Not patterns are useful in pairs or sets of cases that test a feature both enabled and disabled,
   so you can verify that the output does and does not include appropriate code for the feature.
8. Complex tests can be written that test multiple features, but these should be written only after
   the individual features are thoroughly covered.
9. Use YAML literal block scalars for the IDL so whitespace is not compressed in `test.erpc` files.