object dtypes in custom check strategies.

[bernardpazio]

I have been trying to add a data synthesis strategy for shapely geometries but I have run into a couple of issues.

The first is in Engine.numpy_dtype. Which doesn't handle the geometry type which makes sense and should probably be aliased as object.
Then the second issue is in to_numpy_dtype. Which forces any object dtype into a str dtype. I can't really see the reason for this, and it prevents having any custom strategies which return an arbitrary object.

Adding a check for geometry type in Engine.numpy_dtype and removing the object to str cast in to_numpy_dtype allows me to create a custom strategy which returns shapely geometry, and get back a GeoSeries with the correct dtype. Thing is I'm not sure if I am missing something important here, or even the reason the objects are being cast to str.

[cosmicBboy]

Hi @bernardpazio

Long story short, writing custom strategies is currently not well supported in pandera, @bphillips-exos has found a workaround, and there's an issue for adding better support for this: #561

The first is in Engine.numpy_dtype. Which doesn't handle the geometry type

You should check out the geopandas extra: https://pandera.readthedocs.io/en/stable/geopandas.html. It adds first class support for GeoSeries, which supports the geometry type implicitly. (Or you can use gpd.array.GeometryDtype or "geometry" string alias)

You can use this together with the workaround linked above to achieve this.

Then the second issue is in to_numpy_dtype. Which forces any object dtype into a str dtype. I can't really see the reason for this

The main reason is that hypothesis (which pandera uses under the hood for data synthesis), doesn't support object dtypes.

import numpy as np
import hypothesis.extra.numpy as numpy_st

st = numpy_st.from_dtype(np.dtype(object))
# *** hypothesis.errors.InvalidArgument: No strategy inference for object

The behavior of to_numpy_dtype is because pandas uses the object dtype under the hood when you specify str as the dtype in a dataframe/series.

[bernardpazio]

Thanks for your reply.

Yeah already using the geopandas types.

Ok that makes more sense, as you are essentially just trying to map dataframe dtypes straight to the data synthesises that hypothesis has already implemented. So really the root of the issue is that this should be by passed for custom strategies as we do not need to worry about what hypothesis is doing for us as that is on the user themselves.

@bphillips-exos approach does by pass that logic so would work for me, and (potentially) separates the check from the strategy which is not needed in some cases. I think I will adopt that approach. This is something i need for a project I'm currently working on, so i could maybe just pick this up and create a PR. I think it should be handled globally in some way, but that would require some more investigation in how best to do it.

[cosmicBboy]

This is something i need for a project I'm currently working on, so i could maybe just pick this up and create a PR.

Yes, feel free to create a PR with a prototype for supporting geometry data synthesis! I can't promise that it'll make it into the codebase, as I'm currently overhauling the backend validation logic, but your PR would heavily inform the next-gen design of pandera (no major breaking changes to the API, of course).

The overhaul will also touch some of the strategy API, and it should make your custom strategy use-case much easier to support.

For now, here's a working example:

import warnings
from typing import Any, Optional

import geopandas as gpd
import pandas as pd
import pandera as pa
from pandera.extensions import register_check_method
from pandera.typing import Series
from pandera.typing.geopandas import GeoDataFrame, GeoSeries
from shapely.geometry import Point

import hypothesis.strategies as st
import hypothesis.extra.pandas


def pass_through_strategy(
    *_: Any,
    passed_strategy: st.SearchStrategy,
) -> st.SearchStrategy:
    return passed_strategy


@register_check_method(
    statistics=['passed_strategy'],
    strategy=pass_through_strategy,
)
def custom_strategy(*_: Any, passed_strategy: st.SearchStrategy) -> bool:
    return True


class StrategySchema(pa.DataFrameSchema):
    def strategy(
        self,
        *,
        size: Optional[int] = 3,
        n_regex_columns: int = 1,
    ) -> st.SearchStrategy:
        def build_column(column: pa.Column) -> hypothesis.extra.pandas.column:
            custom_strategies = [
                check.strategy()
                for check in column.checks
                if check.name == 'custom_strategy'
            ]
            if not custom_strategies:
                return column.strategy_component()

            # assume that the first check must be a custom strategy
            if len(custom_strategies) > 1:
                raise ValueError("The custom strategy must be the first check")

            custom_strategy, *_ = custom_strategies
            return hypothesis.extra.pandas.column(
                name=column.name,
                elements=custom_strategy,
                dtype=object,
            )

        columns = [build_column(column) for column in self.columns.values()]
        index = hypothesis.extra.pandas.range_indexes(
            min_size=size,
            max_size=size,
        )

        def coerce_dtypes(df):
            dtypes = {name: col.dtype.type for name, col in self.columns.items()}
            return df.astype(dtypes)

        return hypothesis.extra.pandas.data_frames(
            columns=columns,
            index=index,
        ).map(coerce_dtypes)


geo_schema = StrategySchema({
    "geometry": pa.Column(
        gpd.array.GeometryDtype,
        pa.Check.custom_strategy(
            st.builds(Point, st.integers(), st.integers())
        )
    ),
    "region": pa.Column(str, pa.Check.isin(["NA", "SA"])),
})

geo_df = gpd.GeoDataFrame({
    "geometry": [
        Point(0, 0),
        Point(1, 1),
    ],
    "region": ["NA", "SA"]
})

sample = geo_schema.example(size=3).astype({"geometry": "geometry"})
print(sample)
print(sample.dtypes)

You'll probably have to adapt this to break some of the assumptions made in build_column.

This will raise warnings:

ShapelyDeprecationWarning: The array interface is deprecated and will no longer work in Shapely 2.0. Convert the '.coords' to a numpy array instead.
FutureWarning: The input object of type 'Point' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'Point', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`

But I think geopandas will eventually handle this warning.

[bernardpazio]

Thanks for you help. If your undergoing a major revision, probably best to hold off, and I'll just use this work around for now.