Alexandra Kapp

We investigated the utility of five models that create synthetic urban mobility data from raw privacy-sensitive data.

Tl;dr: synthetic trips do not provide the expected high flexibility and utility and should be used with care.

https://dl.acm.org/doi/10.1145/3589132.3625661

Why synthetic data?

Human movement data is highly sensitive, however, data sharing is desirable for many use cases, including city planning or or demand-based transit.
Synthetic data, in this context, is created through models that learn respective distributions from raw data and maintain these. The goal is to create high-utility privacy-friendly synthetic datasets.

How is utility measured?

Typically, such synthetic data models are evaluated by comparing distributions, e.g., the spatial distribution, between raw and synthetic data. The higher the similarity the higher the utility is considered.

However, this approach has shortcomings:
Distributions are typically discretized, e.g., a spatial distribution based on a grid. The resolution of such grids thereby highly influences the conclusion about the maintained utility: a high similarity on a 100m resolution has different implications than on a 1km resolution.
Also: high similarity based on one distribution does not indicate a general high utility. For example, high similarity of spatial distributions does not allow conclusions about temporal distributions.
Single distributions also do not reflect actual real-life use cases.

What did we evaluate?

We selected four tasks that closely reflect real-life tasks that trip data is used for to obtain a more realistic utility evaluation: trip lengths, traffic volume, road preference, and traffic flow at intersections.
We evaluated the utility of five state-of-the-art models, AdaTrace, PrivTrace, DP-Loc, a BiLSTM-based model, and TrajGAIL, using our utility metrics on a dataset comprising approximately 30,000 bicycle trips in Berlin.

First of all, none of the evaluated models provide synthetic data on a level that is fine-granular enough to match the road network. Thus, we included a step of map matching.
Then, we introduced routing-engine-generated trips (like GoogleMaps) as a baseline, as they provide a privacy-friendly way of fine-granular routes to connect a start and an endpoint.

Our results

Out of the five evaluated models, two fail to produce results suitable for our dataset and map-matching approach.
The remaining three models somewhat maintain spatial distribution, one even with differential privacy guarantees. However, all models struggle to produce meaningful sequences of geo-locations with reasonable trip lengths and to model traffic flow at intersections accurately.
It is worth noting that trip data encompasses various relevant characteristics beyond spatial distribution all of which are discarded by these models.
Our results imply that current models fall short in their promise of high utility and flexibility.