Prisma-style ORMs

Prisma, a popular ORM for TypeScript, allows writing queries like (adapted from this example):

const user = await prisma.user.findMany({
  select: {
    name: true,
    email: true,
    posts: true,
  },
});

for which the inferred type will be something like:

{
    email: string;
    name: string | null;
    posts: {
        id: number;
        title: string;
        content: string | null;
        authorId: number | null;
    }[];
}[]

Here, the output type is a combination of both existing information about the type of prisma.user and the type of the argument to findMany. It returns an array of objects containing the properties of user that were requested; one of the requested elements, posts, is a “relation” referencing another model; it has all of its properties fetched but not its relations.

We would like to be able to do something similar in Python, perhaps with a schema defined like:

class Comment:
    id: Property[int]
    name: Property[str]
    poster: Link[User]


class Post:
    id: Property[int]

    title: Property[str]
    content: Property[str]

    comments: MultiLink[Comment]
    author: Link[Comment]


class User:
    id: Property[int]

    name: Property[str]
    email: Property[str]
    posts: Link[Post]

(In Prisma, a code generator generates type definitions based on a prisma schema in its own custom format; you could imagine something similar here, or that the definitions were hand written)

and a call like:

db.select(
    User,
    name=True,
    email=True,
    posts=True,
)

which would have return type list[<User>] where:

class <User>:
    name: str
    email: str
    posts: list[<Post>]

class <Post>:
    id: int
    title: str
    content: str

(Example code for implementing this below.)

Automatically deriving FastAPI CRUD models

In the FastAPI tutorial, they show how to build CRUD endpoints for a simple Hero type. At its heart is a series of class definitions used both to define the database interface and to perform validation/filtering of the data in the endpoint:

class HeroBase(SQLModel):
    name: str = Field(index=True)
    age: int | None = Field(default=None, index=True)


class Hero(HeroBase, table=True):
    id: int | None = Field(default=None, primary_key=True)
    secret_name: str


class HeroPublic(HeroBase):
    id: int


class HeroCreate(HeroBase):
    secret_name: str


class HeroUpdate(HeroBase):
    name: str | None = None
    age: int | None = None
    secret_name: str | None = None

The HeroPublic type is used as the return types of the read endpoint (and is validated while being output, including having extra fields stripped), while HeroCreate and HeroUpdate serve as input types (automatically converted from JSON and validated based on the types, using Pydantic).

Despite the multiple types and duplication here, mechanical rules could be written for deriving these types:

• Public should include all non-“hidden” fields, and the primary key should be made non-optional
• Create should include all fields except the primary key
• Update should include all fields except the primary key, but they should all be made optional and given a default value

With the definition of appropriate helpers, this proposal would allow writing:

class Hero(NewSQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)

    name: str = Field(index=True)
    age: int | None = Field(default=None, index=True)

    secret_name: str = Field(hidden=True)

type HeroPublic = Public[Hero]
type HeroCreate = Create[Hero]
type HeroUpdate = Update[Hero]

Those types, evaluated, would look something like:

class HeroPublic:
    id: int
    name: str
    age: int | None


class HeroCreate:
    name: str
    age: int | None = None
    secret_name: str


class HeroUpdate:
    name: str | None = None
    age: int | None = None
    secret_name: str | None = None

While the implementation of Public, Create, and Update are certainly more complex than duplicating code would be, they perform quite mechanical operations and could be included in the framework library.

A notable feature of this use case is that it depends on performing runtime evaluation of the type annotations. FastAPI uses the Pydantic models to validate and convert to/from JSON for both input and output from endpoints.

Currently it is possible to do the runtime half of this: we could write functions that generate Pydantic models at runtime based on whatever rules we wished. But this is unsatisfying, because we would not be able to properly statically typecheck the functions.

(Example code for implementing this below.)

dataclasses-style method generation

We would additionally like to be able to generate method signatures based on the attributes of an object. The most well-known example of this is probably generating __init__ methods for dataclasses, which we present a simplified example of. (In our test suites, this is merged with the FastAPI-style example above, but it need not be).

This kind of pattern is widespread enough that PEP 681 was created to represent a lowest-common denominator subset of what existing libraries do.

Making it possible for libraries to implement more of these patterns directly in the type system will give better typing without needing further special casing, typechecker plugins, hardcoded support, etc.

(Example code for implementing this below.)

More powerful decorator typing

The typing of decorator functions has long been a pain point in Python typing. The situation was substantially improved by the introducing of ParamSpec in PEP 612, but a number of patterns remain unsupported:

• Adding/removing/modifying a keyword parameter
• Adding/removing/modifying a variable number of parameters. (Though TypeVarTuple is close to being able to support adding and removing, if multiple unpackings were to be allowed, and Pyre implemented a Map operator that allowed modifying multiple.)

This proposal will cover those cases.

NumPy-style broadcasting

One of the motivations for the introduction of TypeVarTuple in PEP 646 is to represent the shapes of multi-dimensional arrays, such as:

x: Array[float, L[480], L[640]] = Array()

The example in that PEP shows how TypeVarTuple can be used to make sure that both sides of an arithmetic operation having matching shapes. Most multi-dimensional array libraries, however, also support broadcasting [1], which allows the mixing of differently shaped data. With this PEP, we can define a Broadcast[A, B] type alias, and then use it as a return type:

class Array[DType, *Shape]:
    def __add__[*Shape2](
        self,
        other: Array[DType, *Shape2]
    ) -> Array[DType, *Broadcast[tuple[*Shape], tuple[*Shape2]]]:
        raise BaseException

(The somewhat clunky syntax of wrapping the TypeVarTuple in another tuple is because typecheckers currently disallow having two TypeVarTuple arguments. A possible improvement would be to allow writing the bare (non-starred or Unpack-ed) variable name to mean its interpretation as a tuple.)

We can then do:

a1: Array[float, L[4], L[1]]
a2: Array[float, L[3]]
a1 + a2  # Array[builtins.float, Literal[4], Literal[3]]

b1: Array[float, int, int]
b2: Array[float, int]
b1 + b2  # Array[builtins.float, int, int]

err1: Array[float, L[4], L[2]]
err2: Array[float, L[3]]
# err1 + err2  # E: Broadcast mismatch: Literal[2], Literal[3]

(Example code for implementing this below.)

Unpack of typevars for **kwargs

A minor proposal that could be split out maybe:

Supporting Unpack of typevars for **kwargs:

def f[K: BaseTypedDict](**kwargs: Unpack[K]) -> K:
    return kwargs

Here BaseTypedDict is defined as:

class BaseTypedDict(typing.TypedDict):
    pass

But any typeddict would be allowed there.

Then, if we had a call like:

x: int
y: list[str]
f(x=x, y=y)

the type inferred for K would be something like:

TypedDict({'x': int, 'y': list[str]})

This is basically a combination of “PEP 692 – Using TypedDict for more precise **kwargs typing” and the behavior of Unpack for *args from “PEP 646 – Variadic Generics”.

When inferring types here, the type checker should infer literal types when possible. This means inferring literal types for arguments that do not appear in the bound, as well as for arguments that do appear in the bound as read-only (TODO: Or maybe we should make whether to do it for extra arguments configurable in the TypedDict serving as the bound somehow? If readonly had been added as a parameter to TypedDict we would use that.)

For each non-required item in the bound that does not have a matching argument provided, then if the item is read-only, it will have its type inferred as Never, to indicate that it was not provided. (This can only be done for read-only items, since non read-only items are invariant.)

This is potentially moderately useful on its own but is being done to support processing **kwargs with type level computation.

—

Extended Callables, take 2

We introduce a Param type the contains all the information about a function param:

class Param[N: str | None, T, Q: ParamQuals = typing.Never]:
    pass

ParamQuals = typing.Literal["*", "**", "default", "keyword"]

type PosParam[N: str | None, T] = Param[N, T, Literal["positional"]]
type PosDefaultParam[N: str | None, T] = Param[N, T, Literal["positional", "default"]]
type DefaultParam[N: str, T] = Param[N, T, Literal["default"]]
type NamedParam[N: str, T] = Param[N, T, Literal["keyword"]]
type NamedDefaultParam[N: str, T] = Param[N, T, Literal["keyword", "default"]]
type ArgsParam[T] = Param[Literal[None], T, Literal["*"]]
type KwargsParam[T] = Param[Literal[None], T, Literal["**"]]

And then, we can represent the type of a function like:

def func(
    a: int,
    /,
    b: int,
    c: int = 0,
    *args: int,
    d: int,
    e: int = 0,
    **kwargs: int
) -> int:
    ...

as (we are omitting the Literal in places):

Callable[
    [
        Param["a", int, "positional"],
        Param["b", int],
        Param["c", int, "default"],
        Param[None, int, "*"],
        Param["d", int, "keyword"],
        Param["e", int, Literal["default", "keyword"]],
        Param[None, int, "**"],
    ],
    int,
]

or, using the type abbreviations we provide:

Callable[
    [
        PosParam["a", int],
        Param["b", int],
        DefaultParam["c", int],
        ArgsParam[int],
        NamedParam["d", int],
        NamedDefaultParam["e", int],
        KwargsParam[int],
    ],
    int,
]

(Rationale discussed below.)

TODO: Should the extended argument list be wrapped in a typing.Parameters[*Params] type (that will also kind of serve as a bound for ParamSpec)?

Grammar specification of the extensions to the type language

Note first that no changes to the Python grammar are being proposed, only to the grammar of what Python expressions are considered as valid types.

(It’s also slightly imprecise to call this a grammar: <bool-operator> refers to any of the names defined in the Boolean Operators section, which might be imported qualified or with some other name)

<type> = ...
     # Type booleans are all valid types too
     | <type-bool>

     # Conditional types
     | <type> if <type-bool> else <type>

     # Types with variadic arguments can have
     # *[... for t in ...] arguments
     | <ident>[<variadic-type-arg> +]

     | GenericCallable[<type>, lambda <args>: <type>]

# Type conditional checks are boolean compositions of
# boolean type operators
<type-bool> =
      <bool-operator>[<type> +]
    | not <type-bool>
    | <type-bool> and <type-bool>
    | <type-bool> or <type-bool>
    | any(<type-bool-for>)
    | all(<type-bool-for>)

<variadic-type-arg> =
      <type> ,
    | * [ <type-for-iter> ] ,


<type-for> = <type> <type-for-iter>+ <type-for-if>*
<type-for-iter> =
      # Iterate over a tuple type
      for <var> in Iter[<type>]
<type-for-if> =
      if <type-bool>

(<type-bool-for> is identical to <type-for> except that the result type is a <type-bool> instead of a <type>.)

There are three core syntactic features introduced: type booleans, conditional types and unpacked comprehension types.

“Generic callables” are also technically a syntactic feature, but are discussed as an operator.

Type operators

In some sections below we write things like Literal[int] to mean “a literal that is of type int”. I don’t think I’m really proposing to add that as a notion, but we could.

class InitField[KwargDict: BaseTypedDict]:
    def __init__(self, **kwargs: typing.Unpack[KwargDict]) -> None:
        ...

    def _get_kwargs(self) -> KwargDict:
        ...

class A:
    foo: int = InitField(default=0, kw_only=True)

Lifting over Unions

Many of the builtin operations are “lifted” over Union.

For example:

Concat[Literal['a'] | Literal['b'], Literal['c'] | Literal['d']] = (
    Literal['ac'] | Literal['ad'] | Literal['bc'] | Literal['bd']
)

When an operation is lifted over union types, we take the cross product of the union elements for each argument position, evaluate the operator for each tuple in the cross product, and then union all of the results together. In Python, the logic looks like:

args_union_els = [get_union_elems(arg) for arg in args]
results = [
    eval_operator(*xs)
    for xs in itertools.product(*args_union_els)
]
if results:
    return Union[*results]
else:
    return Never

Runtime evaluation support

An important goal is supporting runtime evaluation of these computed types. We do not propose to add an official evaluator to the standard library, but intend to release a third-party evaluator library.

While most of the extensions to the type system are “inert” type operator applications, the syntax also includes list iteration and conditionals, which will be automatically evaluated when the __annotate__ method of a class, alias, or function is called.

In order to allow an evaluator library to trigger type evaluation in those cases, we add a new hook to typing:

• special_form_evaluator: This is a ContextVar that holds a callable that will be invoked with a typing._GenericAlias argument when __bool__ is called on a Boolean Operator or __iter__ is called on typing.Iter. The returned value will then have bool or iter called upon it before being returned.

  If set to None (the default), the boolean operators will return False while Iter will evaluate to iter(typing.TypeVarTuple("_IterDummy")). (TODO: Or should it be to iter([])?)

Prisma-style ORMs

First, to support the annotations we saw above, we have a collection of dummy classes with generic types.

class Pointer[T]:
    pass

class Property[T](Pointer[T]):
    pass

class Link[T](Pointer[T]):
    pass

class SingleLink[T](Link[T]):
    pass

class MultiLink[T](Link[T]):
    pass

The select method is where we start seeing new things.

The **kwargs: Unpack[K] is part of this proposal, and allows inferring a TypedDict from keyword args.

Attrs[K] extracts Member types corresponding to every type-annotated attribute of K, while calling NewProtocol with Member arguments constructs a new structural type.

GetName is a getter operator that fetches the name of a Member as a literal type–all of these mechanisms lean very heavily on literal types. GetMemberType gets the type of an attribute from a class.

def select[ModelT, K: typing.BaseTypedDict](
    typ: type[ModelT],
    /,
    **kwargs: Unpack[K],
) -> list[
    typing.NewProtocol[
        *[
            typing.Member[
                typing.GetName[c],
                ConvertField[typing.GetMemberType[ModelT, typing.GetName[c]]],
            ]
            for c in typing.Iter[typing.Attrs[K]]
        ]
    ]
]: ...

ConvertField is our first type helper, and it is a conditional type alias, which decides between two types based on a (limited) subtype-ish check.

In ConvertField, we wish to drop the Property or Link annotation and produce the underlying type, as well as, for links, producing a new target type containing only properties and wrapping MultiLink in a list.

type ConvertField[T] = (
    AdjustLink[PropsOnly[PointerArg[T]], T]
    if typing.IsSub[T, Link]
    else PointerArg[T]
)

PointerArg gets the type argument to Pointer or a subclass.

GetArg[T, Base, I] is one of the core primitives; it fetches the index I type argument to Base from a type T, if T inherits from Base.

(The subtleties of this will be discussed later; in this case, it just grabs the argument to a Pointer).

type PointerArg[T: Pointer] = typing.GetArg[T, Pointer, Literal[0]]

AdjustLink sticks a list around MultiLink, using features we’ve discussed already.

type AdjustLink[Tgt, LinkTy] = (
    list[Tgt] if typing.IsSub[LinkTy, MultiLink] else Tgt
)

And the final helper, PropsOnly[T], generates a new type that contains all the Property attributes of T.

type PropsOnly[T] = typing.NewProtocol[
    *[
        typing.Member[typing.GetName[p], PointerArg[typing.GetType[p]]]
        for p in typing.Iter[typing.Attrs[T]]
        if typing.IsSub[typing.GetType[p], Property]
    ]
]

The full test is in our test suite.

Automatically deriving FastAPI CRUD models

We have a more fully-worked example in our test suite, but here is a possible implementation of just Public

# Extract the default type from an Init field.
# If it is a Field, then we try pulling out the "default" field,
# otherwise we return the type itself.
type GetDefault[Init] = (
    GetFieldItem[Init, Literal["default"]]
    if typing.IsSub[Init, Field]
    else Init
)

# Create takes everything but the primary key and preserves defaults
type Create[T] = typing.NewProtocol[
    *[
        typing.Member[
            typing.GetName[p],
            typing.GetType[p],
            typing.GetQuals[p],
            GetDefault[typing.GetInit[p]],
        ]
        for p in typing.Iter[typing.Attrs[T]]
        if not typing.IsSub[
            Literal[True],
            GetFieldItem[typing.GetInit[p], Literal["primary_key"]],
        ]
    ]
]

The Create type alias creates a new type (via NewProtocol) by iterating over the attributes of the original type. It has access to names, types, qualifiers, and the literal types of initializers (in part through new facilities to handle the extremely common = Field(...) like pattern used here.

Here, we filter out attributes that have primary_key=True in their Field as well as extracting default arguments (which may be either from a default argument to a field or specified directly as an initializer).

dataclasses-style method generation

# Generate the Member field for __init__ for a class
type InitFnType[T] = typing.Member[
    Literal["__init__"],
    Callable[
        [
            typing.Param[Literal["self"], Self],
            *[
                typing.Param[
                    typing.GetName[p],
                    typing.GetType[p],
                    # All arguments are keyword-only
                    # It takes a default if a default is specified in the class
                    Literal["keyword"]
                    if typing.IsSub[
                        GetDefault[typing.GetInit[p]],
                        Never,
                    ]
                    else Literal["keyword", "default"],
                ]
                for p in typing.Iter[typing.Attrs[T]]
            ],
        ],
        None,
    ],
    Literal["ClassVar"],
]
type AddInit[T] = typing.NewProtocol[
    InitFnType[T],
    *[x for x in typing.Iter[typing.Members[T]]],
]

NumPy-style broadcasting

class Array[DType, *Shape]:
    def __add__[*Shape2](
        self, other: Array[DType, *Shape2]
    ) -> Array[DType, *Broadcast[tuple[*Shape], tuple[*Shape2]]]:
        raise BaseException

type MergeOne[T, S] = (
    T
    if typing.Matches[T, S] or typing.Matches[S, Literal[1]]
    else S
    if typing.Matches[T, Literal[1]]
    else typing.RaiseError[Literal["Broadcast mismatch"], T, S]
)

type DropLast[T] = typing.Slice[T, Literal[0], Literal[-1]]
type Last[T] = typing.GetArg[T, tuple, Literal[-1]]

# Matching on Never here is intentional; it prevents infinite
# recursions when T is not a tuple.
type Empty[T] = typing.IsSub[typing.Length[T], Literal[0]]

type Broadcast[T, S] = (
    S
    if typing.Bool[Empty[T]]
    else T
    if typing.Bool[Empty[S]]
    else tuple[
        *Broadcast[DropLast[T], DropLast[S]],
        MergeOne[Last[T], Last[S]],
    ]
)

Extended Callables

We need extended callable support, in order to inspect and produce callables via type-level computation. mypy supports extended callables but they are deprecated in favor of callback protocols.

Unfortunately callback protocols don’t work well for type level computation. (They probably could be made to work, but it would require a separate facility for creating and introspecting methods, which wouldn’t be any simpler.)

I am proposing a fully new extended callable syntax because:

1. The mypy_extensions functions are full no-ops, and we need real runtime objects
2. They use parentheses and not brackets, which really goes against the philosophy here.
3. We can make an API that more nicely matches what we are going to do for inspecting members (We could introduce extended callables that closely mimic the mypy_extensions version though, if something new is a non starter)

Generic Callable

Consider a method with the following signature:

def process[T](self, x: T) -> T if IsSub[T, list] else list[T]:
    ...

The type of the method is generic, and the generic is bound at the method, not the class. We need a way to represent such a generic function both as a programmer might write it for a NewProtocol.

One option that is somewhat appealing but doesn’t work would be to use unbound type variables and let them be generalized:

type Foo = NewProtocol[
    Member[
        Literal["process"],
        Callable[[T], set[T] if IsSub[T, int] else T]
    ]
]

The problem is that this is basically incompatible with runtime evaluation support, since evaluating the alias Foo will need to evaluate the IsSub, and so we will lose one side of the conditional at least. Similar problems will happen when evaluating Members on a class with generic functions. By wrapping the body in a lambda, we can delay evaluation in both of these cases. (The Members case of delaying evaluation works quite nicely for functions with explicit generic annotations. For old-style generics, we’ll probably have to try to evaluate it and then raise an error when we encounter a variable.)

The reason we suggest restricting the use of GenericCallable to the type argument of Member is because impredicative polymorphism (where you you can instantiate type variables with other generic types) and rank-N types (where generics can be bound in nested positions deep inside function types) are cans of worms when combined with type inference [5]. While it would be nice to support, we don’t want to open that can of worms now.

The unbound type variable tuple is so that bounds and defaults and TypeVarTuple-ness can be specified, though maybe we want to come up with a new approach.

Renounce all cares of runtime evaluation

This would have a lot of simplifying features.

TODO: Expand

Support TypeScript style pattern matching in subtype checking

This would almost certainly only be possible if we also decide not to care about runtime evaluation, as above.

Support dot notation to access Members components

Code would read quite a bit nicer if we could write m.name instead of GetName[m]. A general mechanism to support that might look like:

class Member[N: str, T, Q: MemberQuals = typing.Never, D = typing.Never]:
    type name = N
    type tp = T
    type quals = Q
    type definer = D

We considered this but rejected it due to runtime implementation concerns: an expression like Member[Literal["x", int]].name would need to return an object that captures both the content of the type alias while maintaining the _GenericAlias of the applied class so that type variables may be substituted for.

We may have been mistaken about the runtime evaluation difficulty, though: if we required a special base class in order for a type to use this feature, it should work without too much trouble, and without causing any backporting or compatibility problems.

We wouldn’t be able to have the operation lift over unions or the like (unless we were willing to modify __getattr__ for types.UnionType and typing._UnionGenericAlias to do so!)

That just leave semantic and philosophical concerns: it arguably makes the model more complicated, but a lot of code will read much nicer.

TODO: Should we do this?

Replace IsSub with something weaker than “assignable to” checking

Full python typing assignability checking is not fully implementable at runtime (in particular, even if all the typeshed types for the stdlib were made available, checking against protocols will often not be possible, because class attributes may be inferred and have no visible presence at runtime).

As proposed, a runtime evaluator will need to be “best effort”, ideally with the contours of that effort well-documented.

An alternative approach would be to have a weaker predicate as the core primitive.

One possibility would be a “sub-similarity” check: IsSubSimilar would do simple checking of the head of types, essentially, without looking at type parameters. It would not work with protocols. It would still lift over unions and would check literals.

We decided it probably was not a good idea to introduce a new notion that is similar to but not the same as subtyping, and that would need to either have a long and weird name like IsSubSimilar or a misleading short one like IsSub.

Use type operators for conditional and iteration

Instead of writing:

• tt if tb else tf
• *[tres for T in Iter[ttuple]]

we could use type operator forms like:

• Cond[tb, tt, tf]
• UnpackMap[ttuple, lambda T: tres]
• or UnpackMap[ttuple, T, tres] where T must be a declared TypeVar

Boolean operations would likewise become operators (Not, And, etc).

The advantage of this is that constructing a type annotation never needs to do non-trivial computation, and thus we don’t need runtime hooks to support evaluating them.

It would also mean that it would be much easier to extract the raw type annotation. (The lambda form would still be somewhat fiddly. The non-lambda form would be trivial to extract, but requiring the declaration of a TypeVar goes against the grain of recent changes.)

Another advantage is not needing any notion of a special <type-bool> class of types.

The disadvantage is that is that the syntax seems a lot worse. Supporting filtering while mapping would make it even more bad (maybe an extra argument for a filter?).

We can explore other options too if needed.

Make the type-level operations more “strictly-typed”

This proposal is less “strictly-typed” than typescript (strictly-kinded, maybe?).

Typescript has better typechecking at the alias definition site: For P[K], K needs to have keyof P…

We could do potentially better but it would require more machinery.

• KeyOf[T] - literal keys of T
• Member[T], when statically checking a type alias, could be treated as having some type like tuple[Member[KeyOf[T], object, str, ..., ...], ...]
• GetMemberType[T, S: KeyOf[T]] - but this isn’t supported yet. TS supports it.
• We would also need to do context sensitive type bound inference

What exactly are the subtyping (etc) rules for unevaluated types

Because of generic functions, there will be plenty of cases where we can’t evaluate a type operator (because it’s applied to an unresolved type variable), and exactly what the type evaluation rules should be in those cases is somewhat unclear.

Currently, in the proof of concept implementation in mypy, stuck type evaluations implement subtype checking fully invariantly: we check that the operators match and that every operand matches in both arguments invariantly.