codehaus


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: TensorFlow, PyTorch, and manylinux1


On Sat, Dec 15, 2018 at 8:43 PM Philipp Moritz <pcmoritz@xxxxxxxxx> wrote:

> Dear all,
>
> As some of you know, there is a standard in Python called manylinux (
> https://www.python.org/dev/peps/pep-0513/) to package binary executables
> and libraries into a “wheel” in a way that allows the code to be run on a
> wide variety of Linux distributions. This is very convenient for Python
> users, since such libraries can be easily installed via pip.
>
> This standard is also important for a second reason: If many different
> wheels are used together in a single Python process, adhering to manylinux
> ensures that these libraries work together well and don’t trip on each
> other’s toes (this could easily happen if different versions of libstdc++
> are used for example). Therefore *even if support for only a single
> distribution like Ubuntu is desired*, it is important to be manylinux
> compatible to make sure everybody’s wheels work together well.
>
> TensorFlow and PyTorch unfortunately don’t produce manylinux compatible
> wheels. The challenge is due, at least in part, to the need to use
> nvidia-docker to build GPU binaries [10]. This causes various levels of
> pain for the rest of the Python community, see for example [1] [2] [3] [4]
> [5] [6] [7] [8].
>
> The purpose of the e-mail is to get a discussion started on how we can
> make TensorFlow and PyTorch manylinux compliant. There is a new standard in
> the works [9] so hopefully we can discuss what would be necessary to make
> sure TensorFlow and PyTorch can adhere to this standard in the future.
>
> It would make everybody’s lives just a little bit better! Any ideas are
> appreciated.
>
> @soumith: Could you cc the relevant list? I couldn't find a pytorch dev
> mailing list.
>
> Best,
> Philipp.
>
> [1] https://github.com/tensorflow/tensorflow/issues/5033
> [2] https://github.com/tensorflow/tensorflow/issues/8802
> [3] https://github.com/primitiv/primitiv-python/issues/28
> [4] https://github.com/zarr-developers/numcodecs/issues/70
> [5] https://github.com/apache/arrow/pull/3177
> [6] https://github.com/tensorflow/tensorflow/issues/13615
> [7] https://github.com/pytorch/pytorch/issues/8358
> [8] https://github.com/ray-project/ray/issues/2159
> [9] https://www.python.org/dev/peps/pep-0571/
> [10]
> https://github.com/tensorflow/tensorflow/issues/8802#issuecomment-291935940
>
> --
> You received this message because you are subscribed to the Google Groups
> "ray-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ray-dev+unsubscribe@xxxxxxxxxxxxxxxx.
> To post to this group, send email to ray-dev@xxxxxxxxxxxxxxxx.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ray-dev/CAFs1FxUBAag6AThj34twiAB6KY3t5sJSJF3g70K3SvF-%2BzGGgw%40mail.gmail.com
> <https://groups.google.com/d/msgid/ray-dev/CAFs1FxUBAag6AThj34twiAB6KY3t5sJSJF3g70K3SvF-%2BzGGgw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>