Testing and Ensuring Type Annotation Quality

Testing Annotation Accuracy

When creating a package with type annotations, authors may want to validate that the annotations they publish meet their expectations. This is especially important for library authors, for whom the published annotations are part of the public interface to their package.

There are several approaches to this problem, and this document will show a few of them.


For simplicity, we will assume that type-checking is done with mypy. Many of these strategies can be applied to other type-checkers as well.

Testing Using assert_type and --warn-unused-ignores

The idea is to write normal Python files, set aside in a dedicated directory like typing_tests/, which assert certain properties of the type annotations.

assert_type (mypy 0.950 and above) can ensure that the type annotation produces the expected type.

If the following file is under test:

# foo.py
def bar(x: int) -> str:
    return str(x)

then the following file tests foo.py:

from typing_extensions import assert_type

assert_type(bar(42), str)

Clever use of mypy --warn-unused-ignores can be used to check that certain expressions are or are not well-typed. The idea is to have valid expressions along with invalid expressions annotated with type: ignore comments. When mypy --warn-unused-ignores is run on these files, it should pass.

This strategy does not offer strong guarantees about the types under test, but it requires no additional tooling.

If the following file is under test:

# foo.py
def bar(x: int) -> str:
    return str(x)

then the following file tests foo.py:

bar("42")  # type: ignore [arg-type]
bar(y=42)  # type: ignore [call-arg]
r1: str = bar(42)
r2: int = bar(42)  # type: ignore [assignment]

Checking reveal_type output from mypy.api.run

mypy provides a subpackage named api for invoking mypy from a python process. In combination with reveal_type, this can be used to write a function which gets the reveal_type output from an expression. Once that’s obtained, tests can assert strings and regular expression matches against it.

This approach requires writing a set of helpers to provide a good testing experience, and it runs mypy once per test case (which can be slow). However, it builds only on mypy and the test framework of your choice.

The following example could be integrated into a testsuite written in any framework:

import re
from mypy import api

def get_reveal_type_output(filename):
    result = api.run([filename])
    stdout = result[0]
    match = re.search(r'note: Revealed type is "([^"]+)"', stdout)
    assert match is not None
    return match.group(1)

For example, we can use the above to provide a run_reveal_type pytest fixture which generates a temporary file and uses it as the input to get_reveal_type_output:

import os
import pytest

def _in_tmp_path(tmp_path):
    cur = os.getcwd()

def run_reveal_type(tmp_path, _in_tmp_path):
    content_path = tmp_path / "reveal_type_test.py"

    def func(code_snippet, *, preamble = ""):
        content_path.write_text(preamble + f"reveal_type({code_snippet})")
        return get_reveal_type_output("reveal_type_test.py")

    return func

For more details, see the documentation on mypy.api.


pytest-mypy-plugins is a plugin for pytest which defines typing test cases as YAML data. The test cases are run through mypy and the output of reveal_type can be asserted.

This project supports complex typing arrangements like pytest parametrized tests and per-test mypy configuration. It requires that you are using pytest to run your tests, and runs mypy in a subprocess per test case.

This is an example of a parametrized test with pytest-mypy-plugins:

- case: with_params
    - val: 1
      rt: builtins.int
    - val: 1.0
      rt: builtins.float
  main: |
    reveal_type({[ val }})  # N: Revealed type is '{{ rt }}'

Improving Type Completeness

One of the goals of many libraries is to ensure that they are “fully type annotated”, meaning that they provide complete and accurate type annotations for all functions, classes, and objects. Having full annotations is referred to as “type completeness” or “type coverage”.

Here are some tips for increasing the type completeness score for your library:

  • Make type completeness an output of your testing process. Several type checkers have options for generating useful output, warnings, or even reports.

  • If your package includes tests or sample code, consider removing them from the distribution. If there is good reason to include them, consider placing them in a directory that begins with an underscore so they are not considered part of your library’s interface.

  • If your package includes submodules that are meant to be implementation details, rename those files to begin with an underscore.

  • If a symbol is not intended to be part of the library’s interface and is considered an implementation detail, rename it such that it begins with an underscore. It will then be considered private and excluded from the type completeness check.

  • If your package exposes types from other libraries, work with the maintainers of these other libraries to achieve type completeness.


The ways in which different type checkers evaluate and help you achieve better type coverage may differ. Some of the above recommendations may or may not be helpful to you, depending on which type checking tools you use.

mypy disallow options

mypy offers several options which can detect untyped code. More details can be found in the mypy documentation on these options.

Some basic usages which make mypy error on untyped data are:

mypy --disallow-untyped-defs
mypy --disallow-incomplete-defs

pyright type verification

pyright has a special command line flag, --verifytypes, for verifying type completeness. You can learn more about it from the pyright documentation on verifying type completeness.

mypy reports

mypy offers several options options for generating reports on its analysis. See the mypy documentation on report generation for details.