Common to both issues are particular FCID combinations. The FCID is comprised of 3 bytes, 0xDDAAPP, where the letters stand for Domain, Area and Port numbers. The domain portion is unique between switches or VSANs in a fabric, but the same for all ports on a particular switch or VSAN. The area and port numbers are generated by the switch for each WWPN that logs into the switch. Each area can have 256 ports. Cisco begins with area 00 and sequentially assigns port numbers.
The first problem occurs with particular Qlogic adapters if the area portion is the same for the HBA and a zoned target. In this case, the adapter will not recognize the target. Cisco rectifies this issue by assigning a unique area for all Qlogic adapters that exhibit this issue. In our case, the WWPN begins with 21:00:00:1b:32… Each manufacturer device that is identified in the switches Organizationally Unique Identifier (OUI) table will be allocated a unique area for each WWPN. Each of the affected Qlogic adapters receives a unique area and the port number 0x00. So the area will never be the same as the target, but the port number portion will always be 0x00, which leads to the second issue.
First, the 4Gb Brocade switches in Access Gateway mode will not route to a WWPN with the port portion of the FCID being 0x00. Second it will not route to two or more ports with the same port number on the same switch. So Cisco's solution to the first issue is not compatible with routing on these switches. Brocade outlines 4 work arounds for the combined issue. Each recommendation depends on your configuration. The first two options are only available if you don't and will not have target devices connected to the Cisco MDS switch in question. If this is the case, you can undo Cisco's fix by deleting specific entries from the OUI table or putting the switch in flat-fcid allocation mode, which essentially ignores the table. It is probably best to avoid these two work a rounds in order to prevent future problems.
The third option is to basically swap the OUI entries of HBAs and targets. In other words, remove the HBA OUI IDs and add the target device OUI IDs. The HBAs will get sequential port numbers in the same area and the target devices will get unique area IDs resolving both issues. The overall problem with this solution is that the devices need to be connected after the OUI table manipulation. Devices connected before the table update will already have FCIDs assigned. If persistent FCID allocation is enabled, the FCIDs will not change without disconnecting the WWPN and manually deleting them. Otherwise, the VSAN needs to be restarted.
The last option is the option that we chose to implement. Basically, we manually assigned FCIDs to each of the affected HBAs connected via the Brocade switches in Access Gateway mode. We kept track of manually assigned FCIDs in a spreadsheet to ensure that no port numbers were duplicated on a switch. Actually, the switch will not allow you to assign duplicate FCIDs. An FCID is allocated the first time the HBA logs into the switch. The static FCID assignment can be performed in advance if the WWPNs for each HBA are known prior to connection. Otherwise, the host needs to be shutdown so that the FCID can be deleted and the static assignment performed. In my case, after the hosts were shutdown, I had to shut/no shut the interface to release the FCID. Prior, we did verify multipathing on other hosts accessing storage via the same FC port.
In the example below, first I list the free and allocated FCIDs in the VSAN in order to choose a range to assign. Next I check the FCIDs in use on the port that my Qlogic HBA is connected. If they are all the affected Qlogic HBAs, I can tell if they are in use by whether the FCID has been manually assigned. Otherwise, I need to consult the active zoneset. If another server is actively using SAN storage, I check the multipath software on that server. Now, I check to see if the initiator is logged into the switch. In my example, it is not logged in. Next, I proceed to shut the port as if the FCID was in use. Finally, I can delete the existing FCID and add a new one. Afterward, I bring the port back up, verify the new FCID and save the configuration. At this point, I am free to move onto creating aliases and zoning.sh fcdomain address-allocation vsan 5
Free FCIDs: 0xc90028 to 0xc9002a
0xc90066 to 0xc900e3
0xc900e5 to 0xc900ee
0xc900f0 to 0xc900ff
0xc90200 to 0xc902e3
0xc902e5 to 0xc902ee
0xc902f0 to 0xc903ee
0xc903f0 to 0xc904ee
0xc904f0 to 0xc905ee
0xc905f0 to 0xc905ff
0xc90e00 to 0xc90e01
0xc90e03 to 0xc90eee
0xc90ef0 to 0xc90eff
0xc91400 to 0xc914ee
0xc914f0 to 0xc915ee
0xc915f0 to 0xc916ee
0xc916f0 to 0xc919ff
0xc91d00 to 0xc91dee
0xc91df0 to 0xc91eee
0xc91ef0 to 0xc92000
0xc92002
0xc9201c to 0xc920ff
0xc92800 to 0xc929ff
0xc92b00 to 0xc92bff
0xc92d00 to 0xc92eff
0xc93000 to 0xc930ff
0xc93a00 to 0xc93aff
0xc93c00 to 0xc93cff
0xc93f00 to 0xc93fff
0xc94400 to 0xc944ff
0xc94b00 to 0xc9fffe
Assigned FCIDs: 0xc90000 to 0xc90027
0xc9002b to 0xc90065
0xc900e4
0xc900ef
0xc90100 to 0xc901ff
0xc902e4
0xc902ef
0xc903ef
0xc904ef
0xc905ef
0xc90600 to 0xc90dff
0xc90e02
0xc90eef
0xc90f00 to 0xc913ff
0xc914ef
0xc915ef
0xc916ef
0xc91a00 to 0xc91cff
0xc91def
0xc91eef
0xc92001
0xc92003 to 0xc9201b
0xc92100 to 0xc927ff
0xc92a00 to 0xc92aff
0xc92c00 to 0xc92cff
0xc92f00 to 0xc92fff
0xc93100 to 0xc939ff
0xc93b00 to 0xc93bff
0xc93d00 to 0xc93eff
0xc94000 to 0xc943ff
0xc94500 to 0xc94aff
0xc9ffff
Reserved FCIDs: 0xc9ffff
Number of free FCIDs: 52852
Number of assigned FCIDs: 12684
Number of reserved FCIDs: 1
MDS# show flogi database interface fc1/3
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/3 5 0xc90100 21:00:00:1b:32:06:f8:e7 20:00:00:1b:32:06:f8:e7
fc1/3 5 0xc92c00 21:00:00:1b:32:94:99:da 20:00:00:1b:32:94:99:da
Total number of flogi = 2.
MDS# show fcdomain fcid persistent unused vsan 5
Total entries 1.
Persistent FCIDs table contents:
VSAN WWN FCID Mask Used Assignment
---- ----------------------- -------- ----------- ---- ----------
5 21:00:00:1b:32:06:f8:e7 0xc90100 ENTIRE AREA NO DYNAMIC
MDS# conf t
MDS(config)# interface fc1/3
MDS(config-if)# shut
MDS
(config-fcid-db)# fcdomain fcid database
MDS
(config-fcid-db)# no vsan 5 wwn 21:00:00:1b:32:06:f8:e7 fcid 0xc90100
MDS
(config-fcid-db)# vsan 5 wwn 21:00:00:1b:32:06:f8:e7 fcid 0xc92002
MDS
(config-fcid-db)# interface fc1/3
MDS(config-if)# no shut
MDS(config-if)# exit
MDS(config)# exit
MDS# show fcdomain fcid persistent vsan 5
Total entries 188.
Persistent FCIDs table contents:
VSAN WWN FCID Mask Used Assignment
---- ----------------------- -------- ----------- ---- ----------
…
5 21:00:00:1b:32:06:f8:e7 0xc92002 SINGLE FCID YES STATIC
…
MDS# show flogi database interface fc1/3
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/3 5 0xc92002 21:00:00:1b:32:06:f8:e7 20:00:00:1b:32:06:f8:e7
fc1/3 5 0xc92c00 21:00:00:1b:32:94:99:da 20:00:00:1b:32:94:99:da
Total number of flogi = 2.
MDS# copy running-config startup-config
merci beacoup pour ces notes
ReplyDeleteThank you for blogging this, we hit the exact same problem and fixed with the method 3.
ReplyDelete