From George Foo: https://eigenfoo.xyz/prob-prog-frameworks/

Key ingredients of a probabilistic programming framework:

1. Language for specifying a model.
2. Library of probability distributions + facilities to specify arbitrary distributions.
3. Inference algorithm belonging to at least one of MCMC or VI.
4. An optimizer, to compute mode of posterior density.
5. Autodiff library, to compute gradients for items 3 and 4 (inference algo + optimizer)
6. Diagnostics suite to analyze quality of inference.

PyMC3 provides a whole lot of these, alongside ArviZ!