Distributed APIs

Collectives API

Distributed collective operations: all_gather, all_reduce, reduce_scatter, all_to_all