Testing and Ensuring Type Annotation Quality¶
Testing Annotation Accuracy¶
When creating a package with type annotations, authors may want to validate that the annotations they publish meet their expectations. This is especially important for library authors, for whom the published annotations are part of the public interface to their package.
There are several approaches to this problem, and this document will show a few of them.
Note
For simplicity, we will assume that type-checking is done with mypy.
Many of these strategies can be applied to other type-checkers as well.
Testing Using assert_type and --warn-unused-ignores¶
The idea is to write normal Python files, set aside in a dedicated directory like typing_tests/, which assert certain properties
of the type annotations.
assert_type (mypy 0.950 and above) can ensure that the type annotation produces the expected type.
If the following file is under test:
# foo.py
def bar(x: int) -> str:
return str(x)
then the following file tests foo.py:
from typing_extensions import assert_type
assert_type(bar(42), str)
Clever use of mypy --warn-unused-ignores can be used to check that certain
expressions are or are not well-typed. The idea is to have valid expressions along
with invalid expressions annotated with type: ignore comments. When
mypy --warn-unused-ignores is run on these files, it should pass.
This strategy does not offer strong guarantees about the types under test, but it requires no additional tooling.
If the following file is under test:
# foo.py
def bar(x: int) -> str:
return str(x)
then the following file tests foo.py:
bar(42)
bar("42") # type: ignore [arg-type]
bar(y=42) # type: ignore [call-arg]
r1: str = bar(42)
r2: int = bar(42) # type: ignore [assignment]
Checking reveal_type output from mypy.api.run¶
mypy provides a subpackage named api for invoking mypy from a
python process. In combination with reveal_type, this can be used to write
a function which gets the reveal_type output from an expression. Once
that’s obtained, tests can assert strings and regular expression matches
against it.
This approach requires writing a set of helpers to provide a good testing
experience, and it runs mypy once per test case (which can be slow).
However, it builds only on mypy and the test framework of your choice.
The following example could be integrated into a testsuite written in any framework:
import re
from mypy import api
def get_reveal_type_output(filename):
result = api.run([filename])
stdout = result[0]
match = re.search(r'note: Revealed type is "([^"]+)"', stdout)
assert match is not None
return match.group(1)
For example, we can use the above to provide a run_reveal_type pytest
fixture which generates a temporary file and uses it as the input to
get_reveal_type_output:
import os
import pytest
@pytest.fixture
def _in_tmp_path(tmp_path):
cur = os.getcwd()
try:
os.chdir(tmp_path)
yield
finally:
os.chdir(cur)
@pytest.fixture
def run_reveal_type(tmp_path, _in_tmp_path):
content_path = tmp_path / "reveal_type_test.py"
def func(code_snippet, *, preamble = ""):
content_path.write_text(preamble + f"reveal_type({code_snippet})")
return get_reveal_type_output("reveal_type_test.py")
return func
For more details, see the documentation on mypy.api.
pytest-mypy-plugins¶
pytest-mypy-plugins is
a plugin for pytest which defines typing test cases as YAML data.
The test cases are run through mypy and the output of reveal_type can
be asserted.
This project supports complex typing arrangements like pytest parametrized
tests and per-test mypy configuration. It requires that you are using
pytest to run your tests, and runs mypy in a subprocess per test case.
This is an example of a parametrized test with pytest-mypy-plugins:
- case: with_params
parametrized:
- val: 1
rt: builtins.int
- val: 1.0
rt: builtins.float
main: |
reveal_type({[ val }}) # N: Revealed type is '{{ rt }}'
Improving Type Completeness¶
One of the goals of many libraries is to ensure that they are “fully type annotated”, meaning that they provide complete and accurate type annotations for all functions, classes, and objects. Having full annotations is referred to as “type completeness” or “type coverage”.
Here are some tips for increasing the type completeness score for your library:
Make type completeness an output of your testing process. Several type checkers have options for generating useful output, warnings, or even reports.
If your package includes tests or sample code, consider removing them from the distribution. If there is good reason to include them, consider placing them in a directory that begins with an underscore so they are not considered part of your library’s interface.
If your package includes submodules that are meant to be implementation details, rename those files to begin with an underscore.
If a symbol is not intended to be part of the library’s interface and is considered an implementation detail, rename it such that it begins with an underscore. It will then be considered private and excluded from the type completeness check.
If your package exposes types from other libraries, work with the maintainers of these other libraries to achieve type completeness.
Warning
The ways in which different type checkers evaluate and help you achieve better type coverage may differ. Some of the above recommendations may or may not be helpful to you, depending on which type checking tools you use.
mypy disallow options¶
mypy offers several options which can detect untyped code.
More details can be found in the mypy documentation on these options.
Some basic usages which make mypy error on untyped data are:
mypy --disallow-untyped-defs
mypy --disallow-incomplete-defs
pyright type verification¶
pyright has a special command line flag, --verifytypes, for verifying
type completeness. You can learn more about it from
the pyright documentation on verifying type completeness.
mypy reports¶
mypy offers several options options for generating reports on its analysis.
See the mypy documentation on report generation for details.