Copyright 1998-2001, University of Notre Dame.
Authors: Jeffrey M. Squyres, Arun Rodrigues, and Brian Barrett with
         Kinis L. Meyer, M. D. McNally, and Andrew Lumsdaine

This file is part of the Notre Dame LAM implementation of MPI.

You should have received a copy of the License Agreement for the Notre
Dame LAM implementation of MPI along with the software; see the file
LICENSE.  If not, contact Office of Research, University of Notre
Dame, Notre Dame, IN 46556.

Redistribution and use in source and binary forms, with or without
modification, are permitted subject to the conditions specified in the
LICENSE file.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

Additional copyrights may follow.


The alltoall program is a canonical example of MPI usage.  It sends a
message from each MPI rank to each other MPI rank.

Note that we carefully distinguish between the order of sending and
receiving.  It is *not* sufficient to have a single loop where each
rank does a simple "MPI_Send(...); MPI_Recv(...);" -- doing so can
lead to deadlock.  This is a subtlety in the MPI standard --
MPI_Send() *may* or *may not* block.  In LAM, MPI_Send() will block if
a message is too long; it will wait until the message has been started
to be received on the target rank.  Under that size, MPI_Send() will
[most likely] return immediately regardless of what the target rank is
doing.

Note that this is not a LAM-specific issue; every MPI implementation
is coded in this way -- that MPI_Send tries to complete immediately,
but above a certain threshold (usually related a combination of
message size and the cumulative size of unreceived messages to that
target), MPI_Send may block until the target rank starts receiving. */

Finally, notice that this program is actually a poor example of the
all-to-all communication pattern -- there is some degree of
serialization in the sending of messages.  For example, rank N has to
wait for rank 0 to send to it before it can continue.  That is, rank 0
causes a daisy-chain of reactions that allow the rest of the ranks to
continue -- each rank will not "start" until rank 0 contacts it.

Use "make" to compile this example.  Make will use mpicc to compile
the program:

        mpicc -o alltoall alltoall.c 

This program must be run on 2 or more ranks.
