perfSONAR User Q&A and Other Discussion

[Otto J Wittner <>]

Hi David,

So there are (of course) tailored ib-tools which may report bw and 
delay. Thanks for those keywords! They led me to the overview at:
https://docs.nvidia.com/networking/display/MLNXOFEDv461000/InfiniBand+Fabric+Utilities

So "iblinkinfo" and tools from the "perftest-package" 
(https://enterprise-support.nvidia.com/s/article/perftest-package) are 
the ones to consider. Excellent.

BR
O2


> From: "David J. Chaffin" <>
>
> Perfsonar should work over ipoib interfaces but there is little 
reason > to do it.  You'd also need a perfsonar server on every cluster 
node.
> You don't have to search for slow routers and firewalls. The very
> useful program iblinkinfo will tell you every link that's not up to
> par, usually because of cables, and ib_send|read_bw|lat will test the
> links.  They're usually 90-some % of advertised speed.

> --
> David

>> On 02.11.2022 11:02, Otto J Wittner wrote:
> > On 31.10.2022 17:13, Mark Feit wrote:
> > > Otto J Wittner writes:
> > >
> > > Has anyone applied perfSONAR in an Infiniband topology and performed
> > > measurements? If yes, are there any opticals worth being aware of?
> > >
> > > I haven’t personally, but as long as what perfSONAR sees looks like 
> > > IP, there should be no problem with it
> >
> > Yes, right. But to what extent is something interesting/relevant 
> > measured then I wonder...? There seem to be both an IP stack and a 
> > couple of other stacks in Nvidia's Infiniband architecture: 
> > https://docs.nvidia.com/networking/display/MLNXOFEDv494170/Introduction
> > I suspect the uDAPL API is potentially more relevant than the IP API...
> >
> > I was just "fishing" for people how might have look into this.
> >
> > (Mark / Andy, you mentioned a name in this context earlier which I noted 
> > down... but my notes seem to have "hidden themselves")
> >
> > > --Mark