Why does mypy think library imports are missing?

Question

When I run mypy it complains that modules cannot be found:

sal@ahfang:~/workspace/ecs/cx-project-skeleton-repo/src/cx-example-function$ pipenv run python -m mypy .
example_lambda.py:3: error: Cannot find module named 'aws_xray_sdk.core'

But when trying to import that exact same module with the exact same Python interpreter, it seems that the module does exist and is importable.

python 
Python 3.7.3 (default, Apr  3 2019, 05:39:12) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aws_xray_sdk.core
>>>

Other than to force ignore the imports in the mypy.ini file, is there anything I should be doing to help mypy see importable modules that definitely do exist?

Answer 1

So, here's the crux of the issue: mypy does not try type-checking every single module you've imported. Instead, it only attempts to type-check modules that have explicitly opted-in to the typing ecosystem.

Modules can opt-in to the typing ecosystem via two key mechanisms:

1. Add type hints or stubs to their code, and include a file named py.typed within the package they distribute to PyPi (or any other package repository). The presence of this marker makes the package PEP-561-aware. The mypy docs also have more info about PEP-561-aware packages.
2. Alternatively, add stubs to typeshed, the repository of type hints for the standard library and select 3rd party libraries.

The aws_xray_sdk package has done neither of these things, so will be ignored by mypy.

---

This is a bit unfortunate, so what can you do? The Missing imports section of the mypy docs has some detailed recommendations on what to do, but to summarize, you basically have three options which I'll list in order from least to most effort:

1. Just silence the import by manually add # type: ignore comments to each import. You can also add the following section to your mypy config file to have this happen automatically:

   [mypy-aws_xray_sdk]
   ignore_missing_imports = True
   

   Now, anything you import from this module will be treated as being of type Any.

2. Search around and see if anybody has created a third party stubs package for your library: basically, an unofficial (or sometimes semi-official) PEP-561-aware package that only contains type hints. For example, for django, there's django-stubs, for SqlAlchemy, there's sqlalchemy-stubs.

3. Create your own stubs for this library and point to them via the mypy_path option in your mypy config file:

   mypy_path = my_stubs/aws_xray_sdk, my_stubs/some_other_library
   

   These stubs don't have to be complete, necessarily: you can get away with just adding annotations for the few things you're using. (And if they do end up becoming relatively complete, you perhaps look into contributing them back to the open-source community.)

---

Now finally, you may be wondering why mypy behaves this way?

Part of this is because it's not safe in the general case for mypy to just try finding and analyzing the module. Just blindly importing and using packages that are not type-hinting ready can sometimes result in odd type errors, or worse, can result in code incorrectly being marked as type-safe. That is, if you care about type-safety, it's better to be immediately notified that some package you're using has no type hints instead of mypy blindly inferring and smearing Anys all over your code.

Mypy could give a better error message here though, at least most of the time. IMO the fact that it doesn't is largely an oversight. There's some discussion about this in https://github.com/python/mypy/issues/4542.