'\" te .\" Copyright (c) 2004, Sun Microsystems, Inc. All Rights Reserved. .\" The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. .\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. .\" When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] .TH CFGADM_IB 1M "Jul 2, 2008" .SH NAME cfgadm_ib \- InfiniBand hardware specific commands for cfgadm .SH SYNOPSIS .LP .nf \fB/usr/sbin/cfgadm\fR \fB-f\fR [\fB-y\fR | \fB-n\fR] [\fB-v\fR] \fB-c\fR \fIfunction\fR \fIap_id\fR... .fi .LP .nf \fB/usr/sbin/cfgadm\fR [\fB-f\fR] [\fB-y\fR | \fB-n\fR] [\fB-v\fR] [\fB-o\fR \fIhardware_options\fR] \fB-x\fR \fIhardware_function\fR \fIap_id\fR... .fi .LP .nf \fB/usr/sbin/cfgadm\fR \fB-v\fR [\fB-a\fR] [\fB-s\fR \fIlisting_option\fR] [-] [\fIap_id\fR | \fIap_type...\fR] .fi .LP .nf \fB/usr/sbin/cfgadm\fR \fB-v\fR \fB-h\fR [\fIap_id\fR]... .fi .SH DESCRIPTION .sp .LP The InfiniBand hardware specific library \fB/usr/lib/cfgadm/ib.so.1\fR provides the functionality for administering its fabric through the \fBcfgadm\fR(1M) utility. \fBcfgadm\fR operates on attachment points. See \fBcfgadm\fR(1M). .sp .LP An InfiniBand (\fBIB\fR) device is enumerated by the IB nexus driver, \fBib\fR(7D), based on the services from the IB Device Manager (IBDM). .sp .LP The IB nexus driver creates and initializes five types of child device nodes: .RS +4 .TP .ie t \(bu .el o IB Port devices .RE .RS +4 .TP .ie t \(bu .el o IB HCA service (HCA_SVC) devices .RE .RS +4 .TP .ie t \(bu .el o IB Virtual Physical Point of Attachment (VPPA) devices .RE .RS +4 .TP .ie t \(bu .el o I/O Controller (IOC) .RE .RS +4 .TP .ie t \(bu .el o IB Pseudo devices .RE .sp .LP See \fBib\fR(7D) for details on enumeration of IB Port, IB VPPA, and IB HCA_SVC devices. For additional information on IBDM, see \fBibdm\fR(7D). See \fBib\fR(4) for details on IB Pseudo devices. .sp .LP For IB administration, two types of static attachment point are created for the fabric administration as seen by the given host. There is one static attachment point \fBib\fR and all IB devices (either an IOC, Port, VPPA, HCA_SVC, or a Pseudo device) in the fabric are represented as dynamic attachment points based off of it. There is another static attachment point for each Host Channel Adapter (HCA) in the host based on its node Globally Unique Identifier (GUID) value. .sp .LP Attachment points are named through ap_ids. There are two types of ap_ids: logical and physical. The physical ap_id is based on the physical path name. For the IB fabric it is \fB/devices/ib:fabric\fR. The logical ap_id is a shorter, and has a more user friendly name. .sp .LP The static ap_id for the IB fabric is \fBib\fR. The IB devices are dynamic attachment points and have no physical ap_id. The logical ap_id of an IOC contains its GUID, \fBib::\fR\fIIOC-GUID\fR. An example of an IOC ap_id is \fBib::80020123456789a\fR. The logical ap_id of a Pseudo device, see ib(4) for details, is of the format \fBib::\fR\fIdriver_name\fR\fB,\fR\fIunit-address\fR. An example of a pseudo ap_id would be \fBib::sdp,0\fR where "\fBsdp\fR" is the driver name and "\fB0\fR" is its \fIunit-address\fR property. The logical ap_id of Port, VPPA and HCA_SVC device contains its Partition Key (\fIP_Key\fR), \fIPort GUID\fR / \fINode GUID\fR and a communication service-name. The format of ap_id is as below: .sp .ne 2 .na \fBPort device\fR .ad .sp .6 .RS 4n \fBib::\fR\fIPORT_GUID\fR\fB,\fR\fI0\fR\fB,\fR\fIservice-name\fR .RE .sp .ne 2 .na \fBVPPA device\fR .ad .sp .6 .RS 4n \fBib::\fR\fIPORT_GUID\fR\fB,\fR\fIP_Key\fR\fB,\fR\fIservice-name\fR .RE .sp .ne 2 .na \fBHCA_SVC device\fR .ad .sp .6 .RS 4n \fBib::\fR\fIHCA_GUID\fR\fB,\fR0\fB,\fR\fIservicename\fR .RE .sp .LP The Partition Key (\fIP_Key\fR) is \fB0\fR for Port and HCA_SVC devices. The \fIP_Key\fR helps determine the partition to which this port belongs for a VPPA device node. A port might have more than one \fIP_Key\fR. An example of a VPPA device logical ap_id point is \fBib::80245678,ffff,ipib\fR. The \fIport-GUID\fR is \fB80245678\fR, the \fIP_Key\fR is \fB0xffff\fR, and the service name is \fBipib\fR. The service-name information is obtained from the file \fB/kernel/drv/ib.conf\fR which contains service-name strings. The HCA's logical ap_id contains its node GUID value, hca:HCA-GUID. An example is \fBhca:21346543210a987\fR. .sp .LP A listing of the IB attachment points includes information on all IB devices (IOC, VPPA, HCA_SVC, Pseudo, and Port devices seen by the IBDM and the IB nexus driver) in the fabric even if they are not seen by the host and configured for use. .sp .LP The following shows a listing of five IB devices (two IOC, one VPPA, one Port, one HCA_SVC) and one HCA: .sp .in +2 .nf example# cfgadm -al Ap_Id Type Receptacle Occupant Condition hca:21346543210a987 IB-HCA connected configured ok ib IB-FABRIC connected configured ok ib::80020123456789a IB-IOC connected configured ok ib::802abc9876543 IB-IOC connected unconfigured unknown ib::80245678,ffff,ipib IB-VPPA connected configured ok ib::12245678,0,nfs IB-PORT connected configured ok ib::21346543,0,hnfs IB-HCA_SVC connected configured ok ib::sdp,0 IB-PSEUDO connected configured ok .fi .in -2 .sp .sp .LP The ap_id \fBib::802abc9876543\fR shows an IOC device that is not yet configured by the host for use or had been previously offlined by an explicit .sp .in +2 .nf cfgadm \fB-c\fR unconfigure .fi .in -2 .sp .sp .LP operation. The distinction was made by the information displayed under the \fBCondition\fR column. The IB device with a zero P_Key and HCA GUID is a HCA_SVC device. Refer to \fBcfgadm\fR(1M) for more information regarding listing attachment points. .sp .LP The receptacle state for attachment points have the following meanings: .sp .ne 2 .na \fBconnected\fR .ad .sp .6 .RS 4n For an IOC/VPPA/Port/Pseudo/HCA_SVC device, \fBconnected\fR implies that it has been seen by the host. The device might not have been configured for use by Solaris. .sp For a HCA attachment point, \fBconnected\fR implies that it has been configured and is in use. .sp All IB ap_ids are always shown as connected. .RE .sp .LP The occupant state for attachment points have the following meanings: .sp .ne 2 .na \fBconfigured\fR .ad .sp .6 .RS 4n The IB device, and the HCA ap_id, are configured and usable by Solaris. .RE .sp .ne 2 .na \fBunconfigured\fR .ad .sp .6 .RS 4n The IB device at the ap_id was explicitly offlined using \fBcfgadm\fR \fB-c\fR \fBunconfigure\fR, was not successfully configured. This could be because it wasn not successfully configuref for use with Solaris (no driver, or a device problem), or because it was never configured for use by the IB nexus driver. .sp The \fBunconfigured\fR operation is not supported for the HCA attachment point. The IB static apid, \fBib\fR, is shown unconfigured if the system has no IB hardware. .RE .sp .LP The attachment point conditions are: .sp .ne 2 .na \fBfailed\fR .ad .sp .6 .RS 4n Not used. .RE .sp .ne 2 .na \fBfailing\fR .ad .sp .6 .RS 4n Not used. .RE .sp .ne 2 .na \fBok\fR .ad .sp .6 .RS 4n Normal state. Ready for use. .RE .sp .ne 2 .na \fBunknown\fR .ad .sp .6 .RS 4n This state is only valid for IB device that have been probed by IBDM but not yet configured for use by Solaris. It is also shown for devices that have been explicitly offlined by a \fBcfgadm\fR \fB-c\fR \fBunconfigure\fR operation. This condition does not apply to a HCA attachment point. .RE .sp .ne 2 .na \fBunusable\fR .ad .sp .6 .RS 4n Not used. .RE .SH OPTIONS .sp .LP The following options are supported: .sp .ne 2 .na \fB\fB-c\fR \fIfunction\fR\fR .ad .sp .6 .RS 4n The IB hardware specific library supports two generic commands (\fIfunction\fRs). These commands are not supported on the static attachment points (that is, the HCA ap_ids and the IB static \fBib\fR ap_id). .sp The following generic commands are supported: .sp .ne 2 .na \fBconfigure\fR .ad .sp .6 .RS 4n Configure the IB device to be used by Solaris. .RE .sp .ne 2 .na \fBunconfigure\fR .ad .sp .6 .RS 4n Unconfigure the IB device. If successful, \fBcfgadm\fR reports the \fBcondition\fR of this \fBap_id\fR as \fBunknown\fR. .RE .RE .sp .ne 2 .na \fB\fB-f\fR\fR .ad .sp .6 .RS 4n Not supported. .RE .sp .ne 2 .na \fB\fB-h\fR \fIap_id\fR\fR .ad .sp .6 .RS 4n Obtain IB specific help for an IB attachment point. .RE .sp .ne 2 .na \fB\fB-l\fR\fR .ad .sp .6 .RS 4n List the state and condition of IB attachment points. The \fB-l\fR option works as described in \fBcfgadm\fR(1M). .sp When paired with the \fB-a\fR option, displays the dynamic attachment points as well (IOC, VPPA, Port, Pseudo, and HCA_SVC devices). .sp When paired with \fB-v\fR option, displays verbose data about the ap_ids. For an IOC, the \fBInfo\fR field in the .sp .in +2 .nf cfgadm \fB-avl\fR .fi .in -2 .sp output displays the following information: VendorID, IOCDeviceID, DeviceVersion, SubsystemVendorID, SubsystemID, Class, Subclass, Protocol, ProtocolVersion and IDString from the IOCControllerProfile. If the ID string isn't provided then nothing is displayed in its place. These fields are defined in the InfiniBand Specification Volume 1 (http://www.infinibandta.org). .sp For a VPPA, Port, or HCA_SVC device the \fBInfo\fR field in the \fBcfgadm -lav\fR display shows the service name information to which this device is bound. If no such information exists, nothing is displayed. .sp For a Pseudo device \fBcfgadm -alv\fR displays the driver name and its \fBunit-address\fR information. For a HCA the verbose listing displays the VendorID, ProductID of the HCA, number of ports it has, and the PortGUID value of its ports. See \fBEXAMPLES\fR. .RE .sp .ne 2 .na \fB\fB-o\fR \fIhardware_option\fR\fR .ad .sp .6 .RS 4n This option is not currently defined. .RE .sp .ne 2 .na \fB\fB-s\fR \fIlisting_option\fR\fR .ad .sp .6 .RS 4n Attachment points of class ib can be listed by using the select sub-option. Refer to the \fBcfgadm\fR(1M) man page for more information. .RE .sp .ne 2 .na \fB\fB-x\fR \fIhardware_function\fR\fR .ad .sp .6 .RS 4n Perform a hardware specific function. Note that the \fIname\fR can not be more than 4 characters long. .sp The following hardware specific functions are supported: .sp .ne 2 .na \fB\fBadd_service\fR \fB-ocomm\fR\fB=[port|vppa|hca_svc],service=\fR\fIname\fR\fR .ad .sp .6 .RS 4n This hardware specific function is supported on the static \fBIB\fR attachment point. It can be used to add a new service to \fB/kernel/drv/ib.conf\fR file and to update the \fBib\fR(7D) driver. .sp You must use the \fBservice=\fR\fIname\fR option to indicate the new service to be added. You must use the option \fBcomm=[port|vppa|hca_svc]\fR option to add the \fBname\fR service to either \fBport-svc-list\fR or to the \fBhca-svc-list\fR in the \fB/kernel/drv/ib.conf\fR file. See \fBEXAMPLES\fR. .RE .sp .ne 2 .na \fB\fBdelete_service\fR \fB-ocomm\fR\fB=[port|vppa|hca_svc],service=\fR\fIname\fR\fR .ad .sp .6 .RS 4n This hardware specific function is supported on the static \fBIB\fR attachment point only. It can be used to delete an existing service from the \fB/kernel/drv/ib.conf\fR file and also from the \fBib\fR(7D) driver's data base. You must use the \fBservice=name\fR option to indicate which service to delete. You must use the \fBcomm=[port|vppa|hca_svc]\fR option to delete this service from the \fBport-svc-list\fR, \fBvppa-svc-list\fR, or \fBvppa-svc-list\fR of the \fB/kernel/drv/ib.conf\fR file. See \fBEXAMPLES\fR. .RE .sp .ne 2 .na \fB\fBlist_clients\fR\fR .ad .sp .6 .RS 4n Supported on \fBHCA\fR attachment points. Displays all the kernel IB clients using this \fBHCA\fR. It also displays the respective ap_ids of these kernel IB clients and if they have opened an alternate \fBHCA\fR device. See \fBEXAMPLES\fR. .sp \&. .sp If a given kernel IB client does not have a valid ap_id then a \fB-\fR is displayed in that column. .RE .sp .ne 2 .na \fB\fBlist_services\fR\fR .ad .sp .6 .RS 4n This hardware specific function is supported on the static \fBIB\fR attachment point only. It lists all the Port and VPPA services as read from the \fB/kernel/drv/ib.conf\fR file. See \fBEXAMPLES\fR. .RE .sp .ne 2 .na \fB\fBunconfig_clients\fR\fR .ad .sp .6 .RS 4n This hardware specific function is supported on the static HCA attachment point only. It can be used to unconfigure all IB kernel clients of this given HCA. Only IB kernel clients that do not have an alternate HCA are unconfigured. See \fBEXAMPLES\fR. .RE .sp .ne 2 .na \fB\fBupdate_ioc_config\fR\fR .ad .sp .6 .RS 4n This hardware specific function is supported on static ib attachment point and the IOC attachment points. For the \fBib\fR APID, this function updates properties of all the IOC device nodes. For the IOC APID, this function updates the properties of specified IOC device node. This command updates the \fBport-list\fR, \fBport-entries\fR, \fBservice-id\fR, and \fBservice-name\fR IOC node properties . .sp See \fBib\fR(7D). .RE .sp .ne 2 .na \fB\fBupdate_pkey_tbls\fR\fR .ad .sp .6 .RS 4n Supported on the static \fBib\fR attachment point. Updates the \fBPKEY\fR information inside \fBIBTL\fR. \fBIBTL\fR re-reads the \fBP_Key\fR tables for all the ports on each \fBHCA\fR present on the host. .sp See \fBibtl\fR(7D). .RE .RE .SH EXAMPLES .LP \fBExample 1 \fRListing the State and Condition of IB Devices .sp .LP The following command lists the state and condition of IB devices on the system. It only shows the static attachment points. .sp .in +2 .nf example# cfgadm hca:21346543210a987 IB-HCA connected configured ok ib IB-FABRIC connected configured ok .fi .in -2 .sp .sp .LP The \fB-a\fR option lists all attachment points. The following example uses the \fB-a\fR option and lists all attachment points: .sp .in +2 .nf example# cfgadm -a hca:21346543210a987 IB-HCA connected configured ok ib IB-FABRIC connected configured ok ib::80020123456789a IB-IOC connected unconfigured ok ib::80245678,ffff,ipib IB-VPPA connected configured ok ib::21346543,0,hnfs IB-HCA_SVC connected configured ok ib::12245678,0,nfs IB-PORT connected configured ok ib::sdp,0 IB-PSEUDO connected configured ok .fi .in -2 .sp .LP \fBExample 2 \fRListing the Verbose Status of a IB VPPA Device .sp .LP The following command lists the verbose status of a IB VPPA device: .sp .in +2 .nf example# cfgadm -alv ib::80245678,ffff,ipib Ap_Id Receptacle Occupant Condition Information When Type Busy Phys_Id ib::80245678,ffff,ipib connected configured ok ipib unavailable IB-VPPA n /devices/ib:fabric::80245678,ffff,ipib .fi .in -2 .sp .sp .LP A verbose listing of an IOC shows additional information. The following command shows a verbose listing: .sp .in +2 .nf example# cfgadm -alv ib::80020123456789a Ap_Id Receptacle Occupant Condition Information When Type Busy Phys_Id ib::80020123456789a connected configured ok VID: 0xeaea DEVID: 0xeaea VER: 0x5 SUBSYS_VID: 0x0 SUBSYS_ID: 0x0 CLASS: 0xffff SUBCLASS: 0xff PROTO: 0xff PROTOVER: 0x1 ID_STRING: Sample Host Adapter unavailable IB-IOC n /devices/ib:fabric::80020123456789a .fi .in -2 .sp .sp .LP A verbose listing of a Pseudo device shows: .sp .in +2 .nf example# cfgadm -alv ib::sdp,0 Ap_Id Receptacle Occupant Condition Information When Type Busy Phys_Id ib::sdp,0 connected configured ok Driver = "sd p" Unit-address = "0" unavailable IB-PSEUDO n /devices/ib:fabric::sdp,0 .fi .in -2 .sp .sp .LP A verbose listing of a HCA shows: .sp .in +2 .nf example# cfgadm -alv hca:21346543210a987 Ap_Id Receptacle Occupant Condition Information When Type Busy Phys_Id hca:21346543210a987 connected configured ok VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x80245678, port2 GUID: 0x80245679 unavailable IB-HCA n /devices/ib:21346543210a987 .fi .in -2 .sp .sp .LP You can obtain more user-friendly output if you specify these following \fBcfgadm\fR class and field selection options: \fB\fR\fB-s\fR \fB"select=class(ib),cols=ap_id:info"\fR .sp .LP The following command displays only IB ap_ids. The output only includes the \fBap_id\fR and \fBInformation\fR fields. .sp .in +2 .nf # cfgadm -al -s "cols=ap_id:info" ib::80245678,ffff,ipib Ap_Id Information ib::80245678,ffff,ipib ipib .fi .in -2 .sp .LP \fBExample 3 \fRUnconfiguring an Existing IB IOC .sp .LP The following command unconfigures the IB IOC attached to \fBib::80020123456789a\fR, then displays the status of the \fBap_id\fR: .sp .in +2 .nf # cfgadm -c unconfigure ib::80020123456789a Unconfigure the device: /devices/ib:fabric::80020123456789a This operation will suspend activity on the IB device Continue (yes/no)? .fi .in -2 .sp .sp .LP Enter: \fBy\fR .sp .in +2 .nf IB device unconfigured successfully. # cfgadm -al ib::80020123456789a Ap_Id Type Receptacle Occupant Condition ib::80020123456789 IB-IOC connected unconfigured unknown # .fi .in -2 .sp .sp .LP The condition unknown implies that the device node doesn't exist anymore and this IB device's existence is known only to the IB Device Manager. .LP \fBExample 4 \fRConfiguring an IB IOC .sp .LP The following series of commands configures an IB device attached to \fBib::80020123456789a\fR: .sp .in +2 .nf # cfgadm -yc configure ib::80020123456789a # cfgadm -al ib::80020123456789a Ap_Id Type Receptacle Occupant Condition ib::80020123456789a IB-IOC connected configured ok .fi .in -2 .sp .LP \fBExample 5 \fRListing All Kernel IB Clients of a HCA .sp .LP The following command lists all kernel IB clients of an HCA attached to \fBhca:21346543210a987\fR: .sp .in +2 .nf # cfgadm -x list_clients hca:21346543210a987 Attachment Point Clients Alternate HCA ib::80020123456789a ioc1 Yes ib::80245678,ffff,ipib ipib No ib::21346543,0,hnfs hnfs No - ibdm No - ibmf No .fi .in -2 .sp .LP \fBExample 6 \fRAdding a Port Service .sp .LP The following command adds a new Port service called \fBsrp\fR: .sp .in +2 .nf # cfgadm -o comm=port,service=srp -x add_service ib .fi .in -2 .sp .LP \fBExample 7 \fRDeleting a VPPA Service .sp .LP The following command deletes the \fBibd\fR VPPA service \fBibd\fR: .sp .in +2 .nf # cfgadm -o comm=vppa,service=ipib -x delete_service ib .fi .in -2 .sp .LP \fBExample 8 \fRListing Port, VPPA, HCA_SVC Services .sp .LP The following command lists all Port, VPPA, and HCA_SVC services: .sp .in +2 .nf # cfgadm -x list_services ib Port communication services: srp VPPA communication services: ipib nfs HCA_SVC communication services: hnfs .fi .in -2 .sp .LP \fBExample 9 \fRReprobing IOC Devices .sp .LP The following command reprobes all IOC device nodes. .sp .in +2 .nf # cfgadm -x update_ioc_config ib This operation can update properties of IOC devices. Continue (yes/no)? Enter: y # .fi .in -2 .sp .LP \fBExample 10 \fRUnconfiguring All Kernel Clients of a HCA .sp .LP The following command unconfigures all kernel clients of a HCA .sp .in +2 .nf # cfgadm -x unconfig_clients hca:21346543 This operation will unconfigure clients of this HCA. Continue (yes/no)? Enter: y .fi .in -2 .sp .SH FILES .sp .ne 2 .na \fB\fB/usr/lib/cfgadm/ib.so.1\fR\fR .ad .sp .6 .RS 4n Hardware-specific library for generic InfiniBand device administration .RE .SH SEE ALSO .sp .LP \fBcfgadm\fR(1M), \fBconfig_admin\fR(3CFGADM), \fBlibcfgadm\fR(3LIB), \fBib\fR(4), \fBattributes\fR(5), \fBib\fR(7D), \fBibdm\fR(7D), \fBibtl\fR(7D) .sp .LP InfiniBand Specification Volume 1 (http://www.infinibandta.org) .SH NOTES .sp .LP Apart from the listing (\fBcfgadm\fR \fB-l\fR or \fBcfgadm\fR \fB-x\fR \fBlist_clients\fR), only the superuser can execute any functions on an attachment point.