Quantcast
Channel: Clusters and HPC Technology
Viewing all 952 articles
Browse latest View live

Intel MPI support for Amazon's EFA libfabric provider

$
0
0

Hi,

Is anyone working on adding support for Amazon's EFA libfabric provider? Described here:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html

They currently have a build of libfabric that supports their 100Gb/s network with OS bypass. I tried getting Intel MPI to use their libfabric, but it doesn't seem to work. I guess Intel MPI needs to be aware of the specific provider? That surprised me a bit since I thought the point of libfabric would be to provide abstracted access to the fabric.

If support for the EFA provider is in progress, I've got an HPC app that could benefit and would be interested in doing some testing.

 


Compiling for Legacy Xeon E5472 cluster

$
0
0

I have a legacy 128-core cluster with Xeon E5472 processors, still a viable resource. Looking for recommendations on compilers that would be suitable for this platform.

-If Intel, what version/update that is still available for download? I have currently compliled apps with Intel MPI 2019/Update 4 compiler but suspect that it may nor be the best choice.

-Should I go with gcc/gfortran and Openmpi?

Thanks

-Achilles

 

I_MPI_OFI_PROVIDER vs. I_MPI_OFA_ADAPTER_NAME

$
0
0

Hi,

The latter has been removed / deprecated in intelmpi 2019.4 which we are currently testing.

We are using environment modules and up to and including intelmpi 2018.3 we had logic in them which set I_MPI_OFA_ADAPTER_NAME to the appropriate interface based on hostname.

I tried using I_MPI_OFI_PROVIDER the same way, and got an error message:

[0] MPI startup(): libfabric version: 1.7.2a-impi

Abort(1094799) on node 26 (rank 26 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(666)......:
MPID_Init(922).............:
MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available)

I guess that this is either the wrong variable or I am not using it correctly.

I believe the syntax is: I_MPI_OFI_PROVIDER=<name>

Looking at the output of I_MPI_OFI_PROVIDER_DUMP I see stuff like:

name: mlx5_0
name: IB-0xfe80000000000000
prov_name: verbs;ofi_rxm

I'd be grateful for any pointers.

Thanks!

Intel MPI 5.1/2016

$
0
0

I'm setting up a cluster on Azure using RDMA over infiniband. The docs state that the only supported MPI for this is Intel MPI 5.1 (=2016). The images do come with a partial install of this but it's only the runtime, and not any of the compiler wrappers. Is a full version of Intel MPI 5.1/2016 still available anywhere - I can't find it? 2018/19 can be installed via yum which makes life easy!

Alternatively how problematic is building against e.g. impi 2018 then running using the 2016 runtime likely to be? Is that a supported/tested configuration at all?

Are my IMB-MPI1 results correct? they seem to be to high???

$
0
0

Hello,

 

i'm testing on 4 nodes of our hpc-system. They are connected with OmniPath - 100gbit/sec Network.

When i start 

mpirun -genv I_MPI_FABRICS shm:tmi -genv  -host qn12 -n 1 /opt/intel/impi/2018.3.222/bin64/IMB-MPI1 Sendrecv : -host qn11 -n 1 /opt/intel/impi/2018.3.222/bin64/IMB-MPI1

#-----------------------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec
            0         1000         0.88         0.88         0.88         0.00
            1         1000         1.17         1.17         1.17         1.70
            2         1000         1.17         1.17         1.17         3.41
            4         1000         1.15         1.15         1.15         6.97
            8         1000         1.14         1.14         1.14        14.04
           16         1000         1.13         1.13         1.13        28.20
           32         1000         1.14         1.14         1.14        56.04
           64         1000         1.26         1.26         1.26       101.26
          128         1000         1.41         1.41         1.41       181.81
          256         1000         1.35         1.35         1.35       379.82
          512         1000         1.40         1.40         1.40       733.56
         1024         1000         1.70         1.70         1.70      1201.89
         2048         1000         1.88         1.88         1.88      2183.23
         4096         1000         2.67         2.67         2.67      3072.77
         8192         1000         4.16         4.16         4.16      3936.50
        16384         1000         6.24         6.24         6.24      5249.57
        32768         1000        10.00        10.00        10.00      6550.95
        65536          640         5.71         5.71         5.71     22951.32
       131072          320         8.74         8.74         8.74     29979.87
       262144          160        15.23        15.23        15.23     34420.24
       524288           80        33.72        33.72        33.72     31092.59
      1048576           40        88.25        88.25        88.25     23763.59
      2097152           20       217.40       217.40       217.40     19292.85
      4194304           10       419.69       419.81       419.75     19982.04

How i can get 34420 mbyte/sec

 

On the other nodes the result is

 mpirun -genv I_MPI_FABRICS shm:tmi -genv  -host on12 -n 1 /opt/intel/impi/2018.3.222/bin64/IMB-MPI1 Sendrecv : -host on11 -n 1 /opt/intel/impi/2018.3.222/bin64/IMB-MPI1

       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec
            0         1000         1.01         1.01         1.01         0.00
            1         1000         1.21         1.21         1.21         1.65
            2         1000         1.16         1.16         1.16         3.46
            4         1000         1.12         1.12         1.12         7.15
            8         1000         1.13         1.13         1.13        14.16
           16         1000         1.12         1.12         1.12        28.50
           32         1000         1.16         1.16         1.16        55.22
           64         1000         1.16         1.16         1.16       109.97
          128         1000         1.43         1.43         1.43       178.54
          256         1000         1.39         1.39         1.39       367.03
          512         1000         1.48         1.48         1.48       692.40
         1024         1000         1.59         1.59         1.59      1291.33
         2048         1000         1.96         1.96         1.96      2088.74
         4096         1000         2.85         2.85         2.85      2874.57
         8192         1000         4.04         4.04         4.04      4056.40
        16384         1000         6.37         6.37         6.37      5142.52
        32768         1000        10.07        10.07        10.07      6509.22
        65536          640         8.37         8.37         8.37     15665.35
       131072          320        15.00        15.00        15.00     17475.97
       262144          160        30.68        30.68        30.68     17088.92
       524288           80        59.58        59.59        59.58     17596.59
      1048576           40       115.33       115.33       115.33     18184.08
      2097152           20       244.81       244.86       244.83     17129.68
      4194304           10       482.49       482.49       482.49     17386.16

What seems ok.

What is going wrong here?

 

(sorry for my bad english)

 

Best regards, 

Axel

 

 

Running a MPICH program through mpiexec

$
0
0

Hi! If this is the wrong board to post this question, I'll happily move it!

I'm trying to run a prebuilt LAMMPS Windows executable on my Windows machine. According to their documentation (http://packages.lammps.org/windows.html), they're built for MPICH.

When I try to run it using the mpiexec executable that comes with Parallel studio XE, it works perfectly, but it doesn't spread processors correctly. For example, if I launch it using the option -n 4, instead of running a single LAMMPS executable (lmp_mpi) on 4 cores, it runs the same executable 4 times, allocating one core per executable.

Here's my calling method : mpiexec -localonly -n 4 -genv OMP_NUM_THREADS 1 lmp_mpi -in Lammps_test.in

I've tried playing with the value of OMP_NUM_THREADS (as my LAMMPS executable should be using OpenMP), but it hasn't given the desired results. What should I do?

Thank you very much! :)

Antoine.

Which Intel Processors are supported for MPI?

$
0
0

I'm trying to run a code using MPI, but the number of CPUs reported to the code is always 1.  I was originally trying to test on a system with Intel Core Duo CPUs, but was told those CPUs were not supported.  I was then told the Intel Xeon 5600 series CPUs were supported.  We are almost ready to switch to that system, but we're now testing on a system with Intel Xeon Gold 6152 CPUs, and it doesn't work on those CPUs either.  The system we need this to work on has Intel Xeon 5667 CPUs.

So my two questions are:

Will MPI work correctly on Intel Xeon 5667 CPUs?  This is what we need MPI to work on.

Will MPI work correctly on Intel Xeon Gold 6152 CPUs?  If this is supposed to work, then we need to find out why it doesn't work.

As a bonus question, is there somewhere to find out what CPUs MPI does and does not work on?  Like a list?

Changes between Intel MPI 2018 and 2019: is this documented?

$
0
0

We want to move from Intel MPI 2018update3 to 2019update3, but ran into a number of problems, e.g. with fabric selection, hydra options and environment variables not working (or not supported any longer?).  This makes job in our cluster fail, or hang.  In the announcement for the beta program for the 2019 release, one could read that there are e.g. changes in the hydra mechanism, but are those changes documented somewhere?  Like new options, deprecated options, etc ...

One example:  we use LSF as our scheduler, and we set therefore certain I_MPI_HYDRA_* options to reflect this.  However, a multi-node MPI job still uses 'ssh'  as a launcher, where we expect it use 'blaunch'.  This worked nicely in 2018 and earlier versions, but not in 2019. 

Anybody else encountering problems like this?  Or do I need to open a support case?

Thanks!

Bernd


How to select TCP/IP as fabric at runtime with Intel 2019 MPI

$
0
0

Hello,

My apologies: I posted this earlier within another thread, but afterwards decided to submit it as a new query.

I have been struggling for a couple days to figure out the very basic setting of how to correctly instantiate Intel MPI 2019 for use over sockets/TCP. I am able to source mpivars.sh without any parameters and then export FI_PROVIDER=sockets which allows me to compile and run the simple hello world code found all over the place on a single node with n number ranks. However, when I instantiate my environment  in the same way and try to compile PAPI from source, it complains in the configure step that the C compiler (GCC in this case) is not able to create executables. The config.log reveals that it struggles to find libfabric.so.1. Even if I add the libfabrics directory to my LD_LIBRARY_PATH and link to the  libfabrics library, I am not able to build PAPI from source. Additionally, I cannot find good documentation for how to use MPI in the most simple and  basic way - single node and several processes. There is a graphic on several presentations and even software.intel.com/intel-mpi-library which indicates I will be able to choose TCP/IP, among other fabric options, at runtime. I will appreciate your comments and assistance in letting me know the correct way to do this.

Regards,

-Rashawn

Intel mpirun error - AI workload

$
0
0

Hi,

  I tried to run one of my workload model for training on a CentOs cluster for MPI analysis. Please find below the command used and the error is displayed below. Request your help in resolving the issue. 

Commands used 

mpiexec  –ppn 1 -- ./scripts/run_intelcaffe.sh --hostfile ~/mpd.hosts --solver models/intel_optimized_models/multinode/resnet50_8nodes_2s/solver.prototxt --network tcp --netmask enp175s0 --benchmark mpi

mpirun  –ppn 1 –l amplxe-cl -collect hotspots -k sampling-mode=hw -result-dir results -- ./scripts/run_intelcaffe.sh --hostfile ~/mpd.hosts --solver models/intel_optimized_models/multinode/resnet50_8nodes_2s/solver.prototxt --network tcp --netmask enp175s0 --benchmark mpi

I keep getting the following error. 

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 26 PID 72362 RUNNING AT node001
=   EXIT STATUS: 255
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 27 PID 72363 RUNNING AT node001
=   KILLED BY SIGNAL: 9 (Killed)
 

I_MPI_STATS removed

$
0
0

Hi all,

 

I just learned that gathering statistics using I_MPI_STATS is not supported by version 2019 of Intel MPI (https://software.intel.com/en-us/articles/intel-mpi-library-2019-beta-re..., "Removals"). 

 

I found this feature quite useful. Is there now a different way to gather MPI statistics? Will this removal be permanent or might the environment variable be re-introduced in a later version? If none is the case, what is the motivation for removing it?

 

Best,

Christian

AWS, Intel mpi and efa

$
0
0

Im having some issues with the latest Intel mpi and efa on AWS instances.

I installed the Intel MPI from the install script found elsewhere in the support forums.

(https://software.intel.com/sites/default/files/managed/f4/92/install_imp...)

I grabbed the latest libfabric source and built that.

The instance already had AWS's libfabric from the efa setup install but its not in PATH/LD_LIBRARY_PATH for these tests.

[sheistan@compute-041249 ~]$ which mpiexec
/opt/intel/compilers_and_libraries_2019.4.243/linux/mpi/intel64/bin/mpiexec
[sheistan@compute-041249 ~]$ which fi_info
/nasa/libfabric/latest/bin/fi_info
[sheistan@compute-041249 ~]$ fi_info -p efa
provider: efa
    fabric: EFA-fe80::4c2:2aff:fec7:ce80
    domain: efa_0-rdm
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_EFA
provider: efa
    fabric: EFA-fe80::4c2:2aff:fec7:ce80
    domain: efa_0-dgrm
    version: 2.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_EFA
provider: efa;ofi_rxd
    fabric: EFA-fe80::4c2:2aff:fec7:ce80
    domain: efa_0-dgrm
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXD
 

when running on a single node life is good as expected:

[sheistan@compute-041249 ~]$ I_MPI_DEBUG=1 mpiexec --hostfile $PBS_NODEFILE -np 2  ./pi_efa
[0] MPI startup(): libfabric version: 1.9.0a1

[0] MPI startup(): libfabric provider: efa

compute-041249
compute-041249
  pi is approximately:  3.1415926769620652  Relative Error is:  -0.20387909E-05
 Integration Wall Time = 0.005503 Seconds on       2 Processors for n =  10000000
 

but when two nodes are involved it hangs. In this case in a mpi_barrier() call.

[sheistan@compute-041249 ~]$ I_MPI_DEBUG=1 mpiexec --hostfile $PBS_NODEFILE -np 2 -ppn 1 ./pi_efa
[0] MPI startup(): libfabric version: 1.9.0a1

[0] MPI startup(): libfabric provider: efa

compute-041116
compute-041249
^C[mpiexec@compute-041249] Sending Ctrl-C to processes as requested
[mpiexec@compute-041249] Press Ctrl-C again to force abort
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source
pi_efa             0000000000404724  Unknown               Unknown  Unknown
libc-2.26.so       00007F80892447E0  Unknown               Unknown  Unknown
libfabric.so.1.11  00007F8088D5A007  Unknown               Unknown  Unknown
libfabric.so.1.11  00007F8088D5B170  Unknown               Unknown  Unknown
libfabric.so.1.11  00007F8088D0B38D  Unknown               Unknown  Unknown
libfabric.so.1.11  00007F8088D0A72E  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A7CAC90  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A0EBF7B  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A7F5B2F  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A3F42C0  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A052F7C  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A054138  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A17F4E2  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00007F808A05446E  MPI_Barrier           Unknown  Unknown
libmpifort.so.12.  00007F808AF1573C  pmpi_barrier          Unknown  Unknown
pi_efa             0000000000402EF9  Unknown               Unknown  Unknown
pi_efa             0000000000402E52  Unknown               Unknown  Unknown
libc-2.26.so       00007F808923102A  __libc_start_main     Unknown  Unknown
pi_efa             0000000000402D6A  Unknown               Unknown  Unknown
 

Thoughts on something to try or something I missed?

 

thanks

 

s

 

 

 

installing parallel_studio_xe_2019_cluster_edition in a cluster for a non root user with student license

$
0
0

Dear Intel support team,

Please help for below points:

Package "parallel_studio_xe_2019_cluster_edition.tgz" has been downloaded from the Intel student's account of Intel website.  This package needs to be deployed at the HPC cluster of 21 compute nodes and 1 master nodes as a non-root user.  We like to install Intel packages at path /home/apps/intel so that all server can access this lustre file system from the compute nodes and master nodes. 

1. Do we just install package at master node with the standalone license will be sufficient to use intel compilers at compute and master nodes? 

2. Do we need an license manager for it? 

3. Is standalone license will work as floating license if we deploy license manager? if yes, single license means single user can run it at any given time right? 

4. How shall we install intel packages without super user account at RHEL Linux? .

Thanking you 

MPI hangs with multiple processes on same node - Intel AI DevCloud

$
0
0

Hello,

I am using Intel AI DevCloud to run a Deep Reinforcement Learning training, using mpi4py to use several agents to collect data at the same time.

In my framework, I run N jobs (in different nodes) with agents and another job with the optimization algorithm in python.

When I run a single agent in each job, the application works correctly. However, when I try to run more than one agent in the same job (same node), the application hangs.

I do not think the problem is the application itself because it works when there is a single agent per node. Additionally, the same application used to work with multiple agents per node last year, when Intel DevCloud was CentOS.

The application is in this code: https://github.com/alexandremuzio/baselines/blob/71977df495e7840179dd05e...

 

Probably I am not presenting enough data about the problem, so if you need any information regarding MPI stuff I can send in this post.

 

Bad file descriptor for 80+ ranks on single host

$
0
0

Hi,

I am unable to launch a simple MPI application using more than 80 processes on a single host using Intel MPI 2018 update 1 and PBS Pro as job scheduler.

The job is submitted with a script containing:

#PBS -l select=81:ncpus=1
mpiexec.hydra -n 81 -ppn 1 ./a.out

In the call to MPI_Init, the following error is raised on rank 80:

[cli_80]: write_line error; fd=255 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Bad file descriptor
[cli_80]: Unable to write to PMI_fd
[cli_80]: write_line error; fd=255 buf=:cmd=barrier_in
:
system msg for write_line failure : Bad file descriptor
[cli_80]: write_line error; fd=255 buf=:cmd=get_ranks2hosts
:
system msg for write_line failure : Bad file descriptor
[cli_80]: expecting cmd="put_ranks2hosts", got cmd=""
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(805): fail failed
MPID_Init(1743)......: channel initialization failed
MPID_Init(2144)......: PMI_Init returned -1
[cli_80]: write_line error; fd=255 buf=:cmd=abort exitcode=68204815
:
system msg for write_line failure : Bad file descriptor

I looked closer into the issue by running the application through strace. The output for rank 80 shows that the process tries to read from the bash internal file descriptor 255:

<snip>
uname({sys="Linux", node="uvtk", ...})  = 0
sched_getaffinity(0, 128,  { 0, 0, 0, 0, 80000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 128
write(255, "cmd=init pmi_version=1 pmi_subve"..., 40) = -1 EBADF (Bad file descriptor)
write(2, "[cli_80]: ", 10[cli_80]: )              = 10
write(2, "write_line error; fd=255 buf=:cm"..., 72write_line error; fd=255 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
) = 72
write(2, "system msg for write_line failur"..., 56system msg for write_line failure : Bad file descriptor
) = 56
write(2, "[cli_80]: ", 10[cli_80]: )              = 10
write(2, "Unable to write to PMI_fd\n", 26Unable to write to PMI_fd
) = 26
uname({sys="Linux", node="uvtk", ...})  = 0
write(255, "cmd=barrier_in\n", 15)      = -1 EBADF (Bad file descriptor)
write(2, "[cli_80]: ", 10[cli_80]: )              = 10
write(2, "write_line error; fd=255 buf=:cm"..., 47write_line error; fd=255 buf=:cmd=barrier_in
:
) = 47
write(2, "system msg for write_line failur"..., 56system msg for write_line failure : Bad file descriptor
) = 56
write(255, "cmd=get_ranks2hosts\n", 20) = -1 EBADF (Bad file descriptor)
write(2, "[cli_80]: ", 10[cli_80]: )              = 10
write(2, "write_line error; fd=255 buf=:cm"..., 52write_line error; fd=255 buf=:cmd=get_ranks2hosts
:
) = 52
write(2, "system msg for write_line failur"..., 56system msg for write_line failure : Bad file descriptor
) = 56
read(255, 0x7fffe00d6320, 1023)         = -1 EBADF (Bad file descriptor)
write(2, "[cli_80]: ", 10[cli_80]: )              = 10
write(2, "expecting cmd=\"put_ranks2hosts\","..., 44expecting cmd="put_ranks2hosts", got cmd=""
) = 44
write(2, "Fatal error in MPI_Init: Other M"..., 187Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(805): fail failed
MPID_Init(1743)......: channel initialization failed
MPID_Init(2144)......: PMI_Init returned -1
) = 187
write(255, "cmd=abort exitcode=68204815\n", 28) = -1 EBADF (Bad file descriptor)
write(2, "[cli_80]: ", 10[cli_80]: )              = 10
write(2, "write_line error; fd=255 buf=:cm"..., 60write_line error; fd=255 buf=:cmd=abort exitcode=68204815
:
) = 60
write(2, "system msg for write_line failur"..., 56system msg for write_line failure : Bad file descriptor
) = 56
exit_group(68204815)                    = ?

All other ranks communicate with pmi_proxy via a valid file descriptor. For example:

<snip>
uname({sys="Linux", node="uvtk", ...})  = 0
sched_getaffinity(0, 128,  { 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 128
write(16, "cmd=init pmi_version=1 pmi_subve"..., 40) = 40
read(16, "cmd=response_to_init pmi_version"..., 1023) = 57
write(16, "cmd=get_maxes\n", 14)        = 14
read(16, "cmd=maxes kvsname_max=256 keylen"..., 1023) = 56
uname({sys="Linux", node="uvtk", ...})  = 0
write(16, "cmd=barrier_in\n", 15)       = 15
read(16,  <unfinished ...>

Is it possible to specify  a list of available file descriptors used by MPI processes or any other way to circumvent this behavior?

Regards,
Pieter
 


error LNK2001: unresolved external symbol

$
0
0

Hi, I am using the fortran90 with mpi. I followed the steps on this website to link the mpi with visual studio 2017. But it didn't work. It shows error LNK2001 to every MPI commands. Like error LNK2001: unresolved external symbol _MPI_ALLREDUCE; error LNK2001: unresolved external symbol _MPI_BARRIER  .etc. I can not even compile the test.f90 code.

https://software.intel.com/en-us/mpi-developer-guide-windows-configuring...

These attached pictures show my configuration. Is there any problem? Have someone met the same problem before? How can I solve it?

Thanks a lot!

Trouble installing Intel MPI in a docker container

$
0
0

I've been trying to get Intel MPI installed on top of a windowsservercore base image. The MPI installer fails during installation of studio_common_libs_p* sub-installation.

I've attached the logs, but the relevant error is:

WixCreateInternetShortcuts:  Creating IUniformResourceLocatorW shortcut 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Intel Parallel Studio XE 2018\Online Service Center.url' target 'https://supporttickets.intel.com'
WixCreateInternetShortcuts:  Error 0x80040154: failed to create an instance of IUniformResourceLocatorW
WixCreateInternetShortcuts:  Error 0x80040154: failed to create Internet shortcut
CustomAction WixCreateInternetShortcuts returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
 

This looks related to creating a URL shortcut and searching around suggests it's because Explorer is not part of the image. I certainly don't need these shortcuts. All I need is a minimal installation so that I can use MPI from withing the container. I don't need any development support headers or libraries, etc.

Is there a workaround for this issue?

Jeff

MPI shared communication with slided access

$
0
0

Hello

 

I have two questions regarding using mpi shared memory communication

 

 

1) If I have a MPI rank which is the only one that writes to a window, is it necessary to employ mpi_win_lock, and mpi_win_unlock?
I know that my application would never have others trying to write to that window. They only read the content of the window, and I make sure that they read after a MPI_BARRIER, so the content of the window has been updated.

 

 

2) In my application I have one MPI rank, which allocates a shared window that needs to be read by 1:N other MPI ranks.

MPI rank 1 shall only read: rma(1:10) 

MPI rank 2 shall only read rma(11:20) 

MPI rank N shall only read  rma(10*(N-1)+1:10*N)

Currently, all 1 to N ranks  are querying the whole shared window, i.e. the size "10*N" with MPI_WIN_SHARED_QUERY.

 

I am asking if it is possible to apply the MPI_WIN_SHARED_QUERY function such that MPI rank 1 only can access the window from 1:10 and rank 2 from 11:20 etc.

In this way, each rank has a local accessing from 1:10 but they refer to different chunks of the shared window? Is this possible?

 

 

Thanks very much!

Intel MPI 2019 with Mellanox ConnectX-5 / provider=ofi_rxm

$
0
0

Hi, we are currently standing up a new cluster with Mellanox ConnectX-5 adapters. I have found that using openMPI, mvapich2, and intel2018-mpi, we can run MPI jobs on all 960 cores in the cluster, however, using intel2019-mpi we can't get beyond ~300 mpi ranks. If we do, we get the following error for every rank: 

Abort(273768207) on node 650 (rank 650 in comm 0): Fatal error in PMPI_Comm_split: Other MPI error, error stack: 
PMPI_Comm_split(507)...................: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=650, new_comm=0x7911e8) failed 
PMPI_Comm_split(489)...................: 
MPIR_Comm_split_impl(167)..............: 
MPIR_Allgather_intra_auto(145).........: Failure during collective 
MPIR_Allgather_intra_auto(141).........: 
MPIR_Allgather_intra_brucks(115).......: 
MPIC_Sendrecv(344).....................: 
MPID_Isend(662)........................: 
MPID_isend_unsafe(282).................: 
MPIDI_OFI_send_lightweight_request(106): 
(unknown)(): Other MPI error 
---------------------------------------------------------------------------------------------------------- 
This is using the default FI_PROVIDER of ofi_rxm. If we switch to using "verbs", we can run all 960 cores, but tests show an order of magnitude increase in latency and much longer run times. 

We have tried installing our own libfabrics (from the git repo ; also we verified with verbose debugging that we are using this libfabrics) and this behavoir does not change

Is there anything I can change to allow all 960 cores using the default ofi_rxm provider?  Or, is there a way to improve performance using the verbs provider?

For completeness: 
Using MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6-x86_64 ofed 
CentOS 7.6.1810 (kernel = 3.10.0-957.21.3.el7.x86_64) 
Intel Parallel studio version 19.0.4.243 
Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] 

Thanks! 

Eric
 

Fortran+MPI program can't run on OPA when using variable format expressions

$
0
0
The code:
use mpi
implicit none
real*8 send_data(10)
integer ierr,mynode,numnodes,status(10)
integer k

send_data=0.0d0
call mpi_init(ierr)
call mpi_comm_rank(MPI_Comm_World, mynode, ierr)
call mpi_comm_size(MPI_Comm_World, numnodes, ierr)
k=10
if(mynode==0) write(*,100)send_data(:10)
100 format(<k>I4) !can't run with OPA Error
!100 format(10I4) !can run on OPA
call mpi_finalize(ierr)
end

Compiler: 2019.update2 and 2019.update4

Compile command: mpiifort  

OS: CentOS Linux release 7.6.1810

opainfo(10.7.0.0.133) output: 

hfi1_0:1                           PortGID:0xfe80000000000000:00117509010ad11e
   PortState:     Active
   LinkSpeed      Act: 25Gb         En: 25Gb        
   LinkWidth      Act: 4            En: 4           
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 3,4         
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True 
   LID: 0x00000001-0x00000001       SM LID: 0x00000001 SL: 0 
         QSFP Copper,       3m  Hitachi Metals    P/N IQSFP26C-30       Rev 01
   Xmit Data:               1424 MB Pkts:             10048182
   Recv Data:               7270 MB Pkts:             10871417
   Link Quality: 5 (Excellent)

Run: $ mpirun -n 4 -env I_MPI_FABRICS shm:ofi ./a.out

Output:

geospace.430313hfi_userinit: mmap of status page (dabbad0008030000) failed: Operation not permitted
geospace.430313psmi_context_open: hfi_userinit: failed, trying again (1/3)
geospace.430313hfi_userinit: assign_context command failed: Invalid argument
geospace.430313psmi_context_open: hfi_userinit: failed, trying again (2/3)
geospace.430313hfi_userinit: assign_context command failed: Invalid argument
geospace.430313psmi_context_open: hfi_userinit: failed, trying again (3/3)
geospace.430313hfi_userinit: assign_context command failed: Invalid argument
geospace.430313PSM2 can't open hfi unit: -1 (err=23)
Abort(1619087) on node 3 (rank 3 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(666)..........: 
MPID_Init(922).................: 
MPIDI_NM_mpi_init_hook(1070)...: 
MPIDI_OFI_create_endpoint(1751): OFI endpoint open failed (ofi_init.h:1751:MPIDI_OFI_create_endpoint:Invalid argument)

But if it changed from "100 format(<k>I4)" to "100 format(10I4)", it can run well.

Thank you.

Viewing all 952 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>