Python Type Hints from entry to practice

Everyone is already familiar with Python, and even the arguments about its usefulness or uselessness may be tired of reading it. But in any case, as a language that will be added to the college entrance examination subject, it still has its uniqueness. Today we will talk about Python again.

Python is a dynamically strongly typed language

It is mentioned in the book "Smooth Python" that if a language rarely implicitly converts types, it means that it is a strongly typed language. For example, Java, C++, and Python are strongly typed languages.

At the same time, if a language often implicitly converts types, it means that it is a weakly typed language. PHP, JavaScript, and Perl are weakly typed languages.

Of course, the simple example comparison above does not exactly say that Python is a strongly typed language, because Java also supports the addition of integer and string, and Java is a strongly typed language. Therefore, the book "Smooth Python" also has the definition of static typing and dynamic typing: the language that checks the type at compile time is a statically typed language, and the language that checks the type at runtime is a dynamically typed language. Static languages need to declare types (some modern languages use type inference to avoid partial type declarations).

In summary, it is obvious that Python is a dynamically strongly typed language and there is no controversy.

Probe into Type Hints

Type Hints in PEP 484 (Python Enhancement Proposals) [ https://www.python.org/dev/peps/pep-0484/]. It further strengthens the characteristics of Python as a strongly typed language, which was introduced for the first time in Python 3.5. Using Type Hints allows us to write typed Python code, which looks more in line with the strongly typed language style.

Two greeting functions are defined here:

• The common writing is as follows:

name = "world"

def greeting(name):
    return "Hello " + name

greeting(name)

• The wording of adding Type Hints is as follows:

name: str = "world"

def greeting(name: str) -> str:
    return "Hello " + name

greeting(name)

Take PyCharm as an example. In the process of writing code, the IDE will perform type checking on the parameters passed to the function according to the type annotation of the function. If the actual parameter type is found to be inconsistent with the function's formal parameter type annotation, the following prompt will appear:

Type Hints for common data structures

The above shows the usage of Type Hints through a greeting function. Next, we will have a more in-depth study on the writing of Type Hints for common data structures in Python.

Default parameters

Python functions support default parameters. The following is the Type Hints writing method of the default parameters. You only need to write the type between the variable and the default parameter.

def greeting(name: str = "world") -> str:
    return "Hello " + name

greeting()

custom type

For custom types, Type Hints can also be well supported. It is written in the same way as Python's built-in types.

class Student(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age


def student_to_string(s: Student) -> str:
    return f"student name: {s.name}, age: {s.age}."

student_to_string(Student("Tim", 18))

When the type is marked as a custom type, the IDE can also check the type.

container type

When we want to add type annotations to built-in container types, because the type annotation operator [] represents a slicing operation in Python, it will cause a syntax error. Therefore, you cannot directly use the built-in container type as an annotation. You need to import the corresponding container type annotation from the typing module (usually the first letter of the built-in type).

from typing import List, Tuple, Dict

l: List[int] = [1, 2, 3]

t: Tuple[str, ...] = ("a", "b")

d: Dict[str, int] = {
    "a": 1,
    "b": 2,
}

However, the emergence of PEP 585 [ https://www.python.org/dev/peps/pep-0585/] solves this problem. We can directly use Python's built-in types without syntax errors.

l: list[int] = [1, 2, 3]

t: tuple[str, ...] = ("a", "b")

d: dict[str, int] = {
    "a": 1,
    "b": 2,
}

type alias

Some complex nested types are very long to write, and if there are repetitions, it will be painful and the code will not be clean enough.

config: list[tuple[str, int], dict[str, str]] = [
    ("127.0.0.1", 8080),
    {
        "MYSQL_DB": "db",
        "MYSQL_USER": "user",
        "MYSQL_PASS": "pass",
        "MYSQL_HOST": "127.0.0.1",
        "MYSQL_PORT": "3306",
    },
]

def start_server(config: list[tuple[str, int], dict[str, str]]) -> None:
    ...

start_server(config)

At this point, it can be solved by giving an alias to the type, similar to variable naming.

Config = list[tuple[str, int], dict[str, str]]


config: Config = [
    ("127.0.0.1", 8080),
    {
        "MYSQL_DB": "db",
        "MYSQL_USER": "user",
        "MYSQL_PASS": "pass",
        "MYSQL_HOST": "127.0.0.1",
        "MYSQL_PORT": "3306",
    },
]

def start_server(config: Config) -> None:
    ...

start_server(config)

This way the code looks much more comfortable.

Variable parameters

A very flexible aspect of Python functions is that they support variable parameters, and Type Hints also supports type annotations for variable parameters.

def foo(*args: str, **kwargs: int) -> None:
    ...

foo("a", "b", 1, x=2, y="c")

IDE can still check it out.

Generic

Generic support is indispensable when using dynamic languages. Type Hints also provides a variety of solutions for generics.

TypeVar

Use TypeVar to receive any type.

from typing import TypeVar

T = TypeVar("T")

def foo(*args: T, **kwargs: T) -> None:
    ...

foo("a", "b", 1, x=2, y="c")

Union

If you don't want to use generics, but only want to use a few specified types, you can use Union to do it. For example, the definition of concat function only wants to receive str or bytes type.

from typing import Union

T = Union[str, bytes]

def concat(s1: T, s2: T) -> T:
    return s1 + s2

concat("hello", "world")
concat(b"hello", b"world")
concat("hello", b"world")
concat(b"hello", "world")

The IDE check prompt is as follows:

difference between

TypeVar can not only receive generics, it can also be used like Union, just pass in the type range you want to specify as a parameter when instantiating. Unlike Union, functions declared using TypeVar must have the same multi-parameter types, and Union does not impose restrictions.

from typing import TypeVar

T = TypeVar("T", str, bytes)

def concat(s1: T, s2: T) -> T:
    return s1 + s2

concat("hello", "world")
concat(b"hello", b"world")
concat("hello", b"world")

The following are the IDE prompts when using TypeVar as a qualified type:

Optional

Type Hints provides Optional as a short form of Union[X, None], which means that the marked parameter is either of type X or None. Optional[X] is equivalent to Union[X, None].

from typing import Optional, Union

# None => type(None)
def foo(arg: Union[int, None] = None) -> None:
    ...


def foo(arg: Optional[int] = None) -> None:
    ...

Any

Any is a special type that can represent all types. Functions that do not specify return values and parameter types implicitly use Any by default, so the following two greeting functions are equivalent:

from typing import Any

def greeting(name):
    return "Hello " + name


def greeting(name: Any) -> Any:
    return "Hello " + name

When we want to use Type Hints to implement static type writing, but also do not want to lose the unique flexibility of dynamic languages, you can use Any.

When the Any type value is assigned to a more precise type, type checking is not performed. The IDE does not have an error prompt for the following code:

from typing import Any

a: Any = None
a = []  # 动态语言特性
a = 2

s: str = ''
s = a  # Any 类型值赋给更精确的类型

callable objects (functions, classes, etc.)

Any callable type in Python can be annotated with Callable. In the following code annotation, Callable[[int], str], [int] represents the parameter list of the callable type, and str represents the return value.

from typing import Callable

def int_to_str(i: int) -> str:
    return str(i)

def f(fn: Callable[[int], str], i: int) -> str:
    return fn(i)

f(int_to_str, 2)

self-reference

When we need to define a tree structure, we often need to self-reference. When the 16177764c2af21 init method is executed, the Tree type has not been generated, so it cannot be directly annotated like the built-in type str, and the string form "Tree" needs to be used

class Tree(object):
    def __init__(self, left: "Tree" = None, right: "Tree" = None):
        self.left = left
        self.right = right

tree1 = Tree(Tree(), Tree())

The IDE can also check for self-referencing types.

This form can be used not only for self-referencing, but also for pre-referencing.

duck type

A notable feature of Python is its extensive application of duck types. Type Hints provides Protocol to support duck types. When defining a class, you only need to inherit Protocol to declare an interface type. When an interface type annotation is encountered, as long as the received object implements all the methods of the interface type, it can pass the type annotation check, and the IDE will not report an error. The Stream here does not need to explicitly inherit the Interface class, it only needs to implement the close method.

from typing import Protocol

class Interface(Protocol):
    def close(self) -> None:
        ...

# class Stream(Interface):
class Stream:
    def close(self) -> None:
        ...

def close_resource(r: Interface) -> None:
    r.close()

f = open("a.txt")
close_resource(f)

s: Stream = Stream()
close_resource(s)

Since both the file object and the Stream object returned by the built-in open function implement the close method, they can pass the Type Hints check, and the string "s" does not implement the close method, so the IDE will prompt a type error.

Other ways of writing Type Hints

In fact, Type Hints has more than one way of writing. Python also implements two other ways of writing in order to be compatible with the preferences of different people and the migration of old code.

Use comments to write

Let's take a look at an example of the tornado framework (tornado/web.py). It is suitable for making modifications on existing projects, the code has been written, and type annotations need to be added later.

Use a separate file to write (.pyi)

You can create a new .pyi file with the same name as .py in the same directory as the source code, and the IDE can also automatically do type checking. The advantage of this is that you can make no changes to the original code and completely decouple. The disadvantage is that it is equivalent to maintaining two codes at the same time.

Type Hints practice

Basically, the commonly used Type Hints in daily coding have been introduced to everyone, let us take a look at how to apply Type Hints in actual coding.

dataclass-data class

dataclass is a decorator, it can decorate the class, used to add magic methods to the class, such as __init__() and __repr__(), etc. It is in PEP 557[ https://www.python.org/dev /peps/pep-0557/] is defined.

from dataclasses import dataclass, field


@dataclass
class User(object):
    id: int
    name: str
    friends: list[int] = field(default_factory=list)


data = {
    "id": 123,
    "name": "Tim",
}

user = User(**data)
print(user.id, user.name, user.friends)
# > 123 Tim []

The code written above using dataclass is equivalent to the following code:

class User(object):
    def __init__(self, id: int, name: str, friends=None):
        self.id = id
        self.name = name
        self.friends = friends or []


data = {
    "id": 123,
    "name": "Tim",
}

user = User(**data)
print(user.id, user.name, user.friends)
# > 123 Tim []

Note: dataclass does not check the field type.

It can be found that using dataclass to write classes can reduce a lot of repetitive boilerplate code, and the syntax is clearer.

Pydantic

Pydantic is a third-party library based on Python Type Hints. It provides data verification, serialization, and documentation functions. It is a library that is worth learning from. The following is a sample code using Pydantic:

from datetime import datetime
from typing import Optional

from pydantic import BaseModel


class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: Optional[datetime] = None
    friends: list[int] = []


external_data = {
    'id': '123',
    'signup_ts': '2021-09-02 17:00',
    'friends': [1, 2, '3'],
}
user = User(**external_data)

print(user.id)
# > 123
print(repr(user.signup_ts))
# > datetime.datetime(2021, 9, 2, 17, 0)
print(user.friends)
# > [1, 2, 3]
print(user.dict())
"""
{
    'id': 123,
    'signup_ts': datetime.datetime(2021, 9, 2, 17, 0),
    'friends': [1, 2, 3],
    'name': 'John Doe',
}
"""

Note: Pydantic enforces a check on the field type.

Pydantic is written very similar to dataclass, but it does more extra work and also provides a very convenient method such as .dict().

Let's look at an example of Pydantic data validation. When the parameters received by the User class do not meet expectations, a ValidationError exception will be thrown. The exception object provides a .json() method to view the cause of the exception.

from pydantic import ValidationError

try:
    User(signup_ts='broken', friends=[1, 2, 'not number'])
except ValidationError as e:
    print(e.json())
"""
[
  {
    "loc": [
      "id"
    ],
    "msg": "field required",
    "type": "value_error.missing"
  },
  {
    "loc": [
      "signup_ts"
    ],
    "msg": "invalid datetime format",
    "type": "value_error.datetime"
  },
  {
    "loc": [
      "friends",
      2
    ],
    "msg": "value is not a valid integer",
    "type": "type_error.integer"
  }
]
"""

All error information is stored in a list, and the error of each field is stored in a nested dict, where loc identifies the abnormal field and the location of the error, msg is the error message, and type is the error type, so the entire error reason It's clear at a glance.

MySQLHandler

MySQLHandler[ https://github.com/jianghushinian/python-scripts/blob/main/scripts/mysql_handler_type_hints.py] is my pymysql library, which supports the use of with syntax to call the execute method and query results Replacing from tuple to object is also an application of Type Hints.

class MySQLHandler(object):
    """MySQL handler"""

    def __init__(self):
        self.conn = pymysql.connect(
            host=DB_HOST,
            port=DB_PORT,
            user=DB_USER,
            password=DB_PASS,
            database=DB_NAME,
            charset=DB_CHARSET,
            client_flag=CLIENT.MULTI_STATEMENTS,  # execute multi sql statements
        )
        self.cursor = self.conn.cursor()

    def __del__(self):
        self.cursor.close()
        self.conn.close()

    @contextmanager
    def execute(self):
        try:
            yield self.cursor.execute
            self.conn.commit()
        except Exception as e:
            self.conn.rollback()
            logging.exception(e)

    @contextmanager
    def executemany(self):
        try:
            yield self.cursor.executemany
            self.conn.commit()
        except Exception as e:
            self.conn.rollback()
            logging.exception(e)

    def _tuple_to_object(self, data: List[tuple]) -> List[FetchObject]:
        obj_list = []
        attrs = [desc[0] for desc in self.cursor.description]
        for i in data:
            obj = FetchObject()
            for attr, value in zip(attrs, i):
                setattr(obj, attr, value)
            obj_list.append(obj)
        return obj_list

    def fetchone(self) -> Optional[FetchObject]:
        result = self.cursor.fetchone()
        return self._tuple_to_object([result])[0] if result else None

    def fetchmany(self, size: Optional[int] = None) -> Optional[List[FetchObject]]:
        result = self.cursor.fetchmany(size)
        return self._tuple_to_object(result) if result else None

    def fetchall(self) -> Optional[List[FetchObject]]:
        result = self.cursor.fetchall()
        return self._tuple_to_object(result) if result else None

Runtime type check

Type Hints is called Hints instead of Check because it is only a type hint rather than a real check. The usage of Type Hints demonstrated above is actually the function of the IDE to help us complete the type checking, but in fact, the IDE's type checking cannot determine whether an error is reported during code execution, and can only perform the function of syntax checking prompts in the static period. .

To implement mandatory type checking during the code execution phase, we need to write our own code or introduce a third-party library (such as the Pydantic described above). Below I have implemented a dynamic check type at runtime through a type_check function, for your reference:

from inspect import getfullargspec
from functools import wraps
from typing import get_type_hints


def type_check(fn):
    @wraps(fn)
    def wrapper(*args, **kwargs):
        fn_args = getfullargspec(fn)[0]
        kwargs.update(dict(zip(fn_args, args)))
        hints = get_type_hints(fn)
        hints.pop("return", None)
        for name, type_ in hints.items():
            if not isinstance(kwargs[name], type_):
                raise TypeError(f"expected {type_.__name__}, got {type(kwargs[name]).__name__} instead")
        return fn(**kwargs)

    return wrapper


# name: str = "world"
name: int = 2

@type_check
def greeting(name: str) -> str:
    return str(name)

print(greeting(name))
# > TypeError: expected str, got int instead

Just add the type_check decorator to the greeting function to implement runtime type checking.

appendix

If you want to continue to learn more about using Python Type Hints, here are some open source projects I recommend for your reference:

• Pydantic [https://github.com/samuelcolvin/pydantic]
• FastAPI [https://github.com/tiangolo/fastapi]
• Tornado [https://github.com/tornadoweb/tornado]
• Flask [https://github.com/pallets/flask]
• Chia-pool [https://github.com/Chia-Network/pool-reference]
• MySQLHandler [https://github.com/jianghushinian/python-scripts/blob/main/scripts/mysql_handler_type_hints.py]

Recommended reading

TypeScript Enumeration Guide

Practical experience sharing: Use PyO3 to build your Python module