Opened 6 years ago

Last modified 3 years ago

#1526 new feature

MPI_Alloc_mem registered memory

Reported by: dinan Owned by:
Priority: major Milestone: future
Component: mpich Keywords: memory registration, buffer pinning
Cc: jhammond

Description

Currently, MPICH2's MPI_Alloc_mem does not accept any info keys and passes through to libc malloc. It would be great if we can add support for passing through to a device allocator that allocates registered memory. If it's not always the desired behavior, the device allocator could be selected through the use of an info key.

Change History (5)

comment:1 Changed 6 years ago by gropp

This isn't quite correct. MPI_Alloc_mem accepts an info key and passes the pointer to the info structure to MPID_Alloc_mem. However, the current CH3 implementation simply calls MPIU_Malloc. No change is needed in either the top level MPI code or the MPID API. Thus, this doesn't apply to non-CH3 device implementations. In the case of CH3, the challenge is to maintain the simplicity of the CH3 interface, since CH3 is supposed to provide a starting point for simple implementations (that is its *primary* requirement). At some point, it may make more sense to implement a revision of MPID, rather than keep pushing CH3 away from its design point.

comment:2 Changed 6 years ago by jhammond

I think that any network device that supports or requires special memory allocation is not simple and therefore outside the scope of the CH3 design point.

I agree with Bill that it makes sense to develop a new device interface that is oriented towards non-simple networks as are found in many supercomputers.

The upside of a new device is that it would hopefully encourage vendors not to either (1) make a new device for a single machine (e.g. dcmfd), or (2) bootstrap CH3 in such a way as to hinder performance.

The downside is that this would seem to approximately double the amount of work required by the MPICH2.

comment:3 Changed 6 years ago by buntinas

Jeff, the changes to ch3 were not very large. I've implemented this in the one-sided communications branch already.

comment:4 Changed 6 years ago by jhammond

I guess I was thinking that things got less simple beyond registered memory, i.e. as one attempts to design first-class support for RDMA-oriented networks.

comment:5 Changed 3 years ago by jhammond

  • Cc jhammond added; jhammond@…, buntinas@… removed
Note: See TracTickets for help on using tickets.