cmvr_head/packages/erpc-1.14.0/erpcgen/test/readme.md

erpcgen test system

This file documents the parser and output test system for erpcgen.

This test system is built on py.test using its extensive plugin hooks.

Setup

Python 3.6+ is required.

py.test(Version 5.0.0-) and pyYAML are required to run the tests. These can be installed via pip.

pip3 install pytest pyyaml

Make sure you have an erpcgen executable available. Either build it via the makefiles, or build the VisualStudio project.

Running

Run the tests by running the py.test executable. The tests can be run from either the erpc/erpcgen/ or erpc/erpcgen/test/ directories. The tests can be also run from root folder defining directory with tests as next parameter.

Run all tests:

$ pytest erpcgen/test

Run one test

$ pytest erpcgen/test/test_union_py.yml

Use the --help option of py.test to see all available command line options. Useful options are -s to turn off output capturing and -v to list each test case by name as it runs. The -k option lets you filter the test cases that are executed by a contains match on the name.

There is one erpcgen-specific command line option, --erpcgen-verbosity or --ev. This lets you set the number of -v arguments that are passed to erpcgen calls. Specify an integer from 1 to 3, such as --ev=3. Default verbosity is 0.

Config

The tests use a config file. The standard config is erpc/erpcgen/test/config.py. It tries to figure out the erpcgen executable location based on the OS. If you need to override this, you can create a config_local.py that defines the config variables listed below.

Config variables:

• CC = C compiler path, defaults to "gcc"
• CXX = C++ compiler path, defaults to "g++"
• ERPCGEN = absolute path to the erpcgen executable
• RUN_KEEP_COUNT = number of historical test runs to retain, not including the most recent run

The ERPCGEN and CC config variables can also be overridden with an environment variables of the same name.

Output

The files associated with test runs are stored in numbered directories under erpc/erpcgen/test/runs/. By default, only the most recent 4 test runs are retained. See the RUN_KEEP_COUNT config variable. A symlink called 'latest' is maintained to point at the most recent test run.

Test cases are stored in a hierarchy of directories under the test run directory:

erpc/erpcgen/test/runs/<run#>/<yaml>/<spec>/[<params>]

The "<params>" directory is added only if the test is parametrized. It will have a name based on the unique set of parameter values for the test case.

Under the test case directory, the test IDL is stored in a test.erpc file. Generated files are written by erpcgen into an output/ subdirectory. Because the IDL file is named "test", the output files names will by default also begin with "test", unless this is overridden through erpcgen arguments or annotations.

If the erpcgen verbosity (see the --erpcgen-verbosity command line option above) is set to a non-zero level, then the stdout output from erpcgen is captured in a erpcgen.out file in the test case directory. This is useful for debugging issues.

For C language test cases, an objects/ directory is created in the test case directory. It holds compilation test inputs and outputs.

Test specs

All .yml files in erpc/erpcgen/test/ whose names begin with "test" are processed for test specifications. Spec files can also be placed into subdirectories of erpc/erpcgen/test/, as long as the subdirectory's name starts with "test".

Multiple test specs can be defined in each YAML file, separated by three dashes ("---"). (These are multiple top-level YAML documents). Place the dashes before each test spec, not after. Each test spec is a dictionary.

Test spec keys:

• idl = input erpcgen IDL
• name = short test name
• desc = description of the test case
• lang = output language (c or py); defaults to c
• params = parametrization, described below
• args = additional erpcgen command line arguments
• jira = JIRA issue key, for reference only
• (output filenames)

The only required key is idl.

Test specs are turned into one or more test cases via parametrization. If the params key is not included in the spec, parametrization is disabled and only one test case will be produced. See below for more about parametrization.

Each test case has its own directory under the main directory for the test run. When a test case is run, the IDL is first written to a test.erpc file within the case directory. Then erpcgen is executed with the output directory set to the output/ subdirectory of the case directory. C code is generated by default; this can be overridden with the lang spec key.

Filenames listed in the spec dictionary are relative to the output/ directory of a test run. Under the filenames are lists of test patterns that must all match against the file under which they are listed for the test case to pass.

A basic test specification looks like this

---
name: quit_fn
idl: |
    interface xyz {
    oneway quit()
    }    
test.h:
    - void quit(void)

This spec says to generate C output for the given IDL. It will match the string "void quit(void)" against the test.h output file. If that string is found, the test will pass.

Test patterns

Under each output filename in the test spec dictionary is a list of test patterns.

Test patterns must be listed in the order they will appear in the file. Each successive pattern will start searching from the point where the last pattern matched. If you need to match multiple, out of order, patterns then you can have the same output file listed more than once.

Patterns, by default, use simple literal string comparison.

Note that you must be aware of the YAML flow notation syntax. If you see YAML parse errors when you run the tests, this is likely the cause. Patterns consisting of only one brace must be put in single or double quotes. The same applies to patterns beginning with an open or close brace.

Regular expressions

A pattern case be made a regular expression by turning it into a dict with a single re key. For example:

- test.h:
    - re: somevar[a-z]+

You must remember to escape all regex control characters with a backslash if you wish to match them literally.

Literal patterns are actually converted to regexs internally, with appropriate escape of control characters. Any space characters in literal patterns are replaced with "\s*", which will match zero or more spaces. This is to make the patterns a little more resilient to minor changes in the output formatting.

Not patterns

Sometime you want to make sure that a certain string is not present in the output. This can be achieved by making a pattern a dict with a single not or not_re key. The former treats the pattern as a literal match while the latter a regex.

Example:

- test/common.py:
    - not: class Foo
    - A = 1

Not patterns are searched for slightly differently than other patterns. Instead of searching to the end of the file, not patterns are searched for within the range of the match positions of the positive patterns before and after the group of not patterns, or the begin and end of the file if there is not a positive pattern before or after the not pattern group. This limits the search range of not patterns, and increases their effectiveness.

In the example above, the "not: class Foo" pattern is searched for from the start of the test/common.py file to the position where the "A = 1" pattern successfully matched.

Consider this example:

- test/common.py:
    - import erpc
    - not: class Foo
    - not_re: A = [0-9]+
    - B = 1

Here, both not pattern searches range from the last matching character of the positive "import erpc" pattern to the first matching character of the "B = 1" pattern. This example also shows a not_re pattern that requires a regular expression to not match within the search range.

Parametrization

If the test spec dictionary has a params key, then the spec will be parametrized. The params key must have a dictionary value with parameter names for keys. Under each of the parameter names must be a list of values.

Here's an example valid params dict and IDL that uses the params:

    params:
        type:
            - string
            - binary
        length:
            - 1
            - 8
    idl: |
        struct Foo {
            {type}[{length}] my_array
        }        

A parametrized test generates test cases for all permutations of parameter values. In the above examples, 4 test cases would be created for the 4 combinations of type and length param values. Each param only has one value per test case. However, you can set an individual parameter value to a list or dict (YAML flow notation works nicely for this). If you only have a single parameter, you would get one test case for each value listed for that parameter.

The IDL and all test patterns will be formatted using parameter values. The Python str.format() method is used. A substitution of the form "{param_name}" in the IDL or a test pattern will be replaced with the parameter's value for that test case.

If a parameter value is a list or dict, you can substitute the elements by using index brackets: "{param_name[3]}" or "{param_name[dict_key]}". Note that a dict key used in index brackets is not placed in quotes. See the str.format() docs for full details of the format options.

Normally, str.format() will raise exceptions for open or close braces by themselves. Yet, the IDL and C code frequently use braces. So, to simplify usage, the test spec strings are pre-processed to convert isolated braces into the double braces required by str.format(). An open or close brace must be followed by at least one whitespace character to be properly recognized and converted.

If-then patterns

To make test specs even more flexible when using parametrization, simple if-then expressions are allowed in the test patterns. This is done by making a test pattern a dictionary with if, then, and optional else keys. The value of the if key is a Python Boolean expression, which is formatted with str.format() (like all test pattern strings) prior to evaluation. The namespace of the if expression has all parameter names set to the values for the test case.

The value of the then key is a list of test patterns which will be used if the if predicate evaluates to true. If an else key is present, its value is a list of test patterns that will be used only if the if predicate evaluates to false.

Example if-then pattern:

test/common.py:
    - if: type=="string"    # type is a param
    then:
        - start_read_list
        - re: read_[a-z]+     # regex test pattern
        - end_read_list
    else:
        - read_string

If-then patterns can be nested, and include regex and not patterns as shown in the example above. However, "else if" expressions are not supported. If you need an "else if", you can use multiple if-then patterns.

Compilation testing

The erpcgen output from each of the test cases is run through the C/C++ compiler or Python. This is done only to test syntax. The code is not linked into an executable.

C language tests have an objects/ directory created in the test case directory. Source files named main_c.c and main_cxx.cpp are written into this directory, and are included in the compile. These files test that the common generated header file can successfully be included by both the C and C++ compilers. The objects/ directory also contains .o files written by the compiler.

For Python test cases, the generated package is loaded and compiled. Each of the modules within the package are also loaded and compiled explicitly. Loading is done directly within the Python executable running py.test.

The CC config variable is used for the path to the C/C++ compiler. The default value is simply "gcc". You can override this with a config_local.py file, or an environment variable.

Compilation testing is not currently performed on Windows systems.

Tips for writing test specs

1. Keep each test spec simple and focused.
2. Test only one feature at a time.
3. Test patterns should focus on the key identifiers and structural elements; don't necessarily make a pattern match an entire output line. This will help keep the tests more flexible and resilient to generator code changes, yet still verify that the important parts of the output are present.
4. Use the fact that patterns must match in order with the file to improve the accuracy of your tests, by adding "anchor" patterns before and after your patterns to ensure they match within a certain area of the output.
5. Use parametrization to produce many test cases with few test specs.
6. If-then patterns can expand the number of cases you can generate with a single spec.
7. Not patterns are useful in pairs or sets of cases that test a feature both enabled and disabled, so you can verify that the output does and does not include appropriate code for the feature.
8. Complex tests can be written that test multiple features, but these should be written only after the individual features are thoroughly covered.
9. Use YAML literal block scalars for the IDL so whitespace is not compressed in test.erpc files.