Cisco IP Device Tracking - the worlds gone mad

IPDT uses ARP inspection to maintain a database of MAC/IP per VLAN off every switchport. This information is used by features that have dependencies on it such ... Togglenavigation theworldsgonemad automation networking security virtual/cloud SEARCH EverthoughtabouthowACSgetsanendusersIPorhowwhenshowinganinterfacesauthenticationsessionsithadtheIPofthehostattached?ThisallstemsfromIPDeviceTracking. IonlyrecentlycameacrossitwhentroubleshootinganissuewehadwithwindowsmachinesnotgettingaDHCPaddressduecollisiondetectioninvolvingthe0.0.0.0address.Althoughtherearelotsofpostsaboutpeoplehavingasimilarissueandtheworkarounds,Icouldn’tfindmuchinformationontheexactreasonwhythishappened.Thispostisdesignedtogivemorereasoningonwhythishappens. TableOfContents IPDTOperation FactsaboutIPDT CollisionDetection(RFC5227) HowdoesthisrelatetoIPDT IPDTOperation IPDTusesARPinspectiontomaintainadatabaseofMAC/IPperVLANoffeveryswitchport.Thisinformationisusedbyfeaturesthathavedependenciesonitsuchas802.1x,MAB(ACS&ISE),Netflow,Trustsecandweb-auth.ForexamplewithMAB,theportisfirstauthenticatedtheninasubsequentRADIUSupdatepacketthedevicetrackinginfo(IPofdevice)ispassedontoACS. AsitusesARP,evenifyoudon’tusethisfeatureforanythingelseitisagreatwaytocheckARPentrieswithouthavingtologintothedevicethatisthegatewayforthesubnet. OnearliercodeIPDTisapparentlydisabledbydefault(notalwaystrue),butfrom15.2(1)E upwardsitcan`tbeenabledordisabledmanually.Itisenableddependentonthefeaturesthatuseit,soifafeaturethatreliesonitisenabled,IPDTisalsoenabled. YoucanseeonaninterfacewhichfeaturesuseIPDT,sotodisableitforthatinterfaceyouneedtodisablethosefeatureonthatinterface. Switch#showipdevicetrackinginterfacegig1/0/9 -------------------------------------------- InterfaceGigabitEthernet1/0/9is:STANDALONE IPDeviceTracking=Disabled IPDeviceTrackingProbeCount=3 IPDeviceTrackingProbeInterval=180000 IPv6DeviceTrackingClientRegisteredHandle:75 IPDeviceTrackingEnabledFeatures: HOST_TRACK_CLIENT_ATTACHMENT HOST_TRACK_CLIENT_SM ThisseemsabitshortcomingbyCiscoastheyrecommendnottoenableIPDTontrunklinks,butifyouuseoneofthesefeaturesonatrunklinkyoucan’tdisableit.Ciscorecommendationsregardstrunksarebasedonitpotentiallyeffectingswitchperformance: TheIPDTdatabaseforatrunkportwillbeverylargeasyouaretrackingthewholeARPtable. Probesaresentevery30secondssoalargetablewillincreasenetworktraffic.Activitiesofhostsconnectedindirectlyoveratrunkmaynotcometothisswitchatallbutwillstillbeprobed. ProbessentbyremoteswitchesconnectedtothelocalswitchviatrunkportsmaybereceivedthatcouldincreasethechanceoftheduplicateIPAddressissueintroducedbyIPDTprobes. TheIPDTtableissyncedbetweenactiveandstandbystackmembers,sotrunkportflapswillcauseentryinsertion/deletionresultinginalargenumberofnotificationflooding. FactsaboutIPDT IfaninterfacegoesdownthestateofentriesinthetablearechangedtoINACTIVE.ThisalsoappliesifaPCattachedtoaCiscophoneisshutdown,theentryimmediatelybecomesINACTIVE. Allentries(ACTIVEandINACTIVE)remainintheIPdevicetrackingtableuntilthemaximumtablelimitisreached(switchdependent),afterwhichtheoldestentrywillthenbereplaced. Thedefaultprobeinterval(agingtimer)is30seconds.Theidletimeoutisactuallytheprobeintervalmultipliedbythenumberofprobeentries(3),so90seconds(idletimer). Theswitchwillsendaprobeevery30seconds(agingtime)toeveryportthatithasanACTIVEentry. IftheswitchreceivesanARPpacket(request,replyorGARP)onaport,itresetstheagingtimebackto30secondsforthatport. IftheagingtimeexpirestheswitchsendsanARPprobetoverifythatthehostispresent.Ifthehostispresent,itsendsaresponsetotheswitchandtheswitchupdatesitsentry. IfaninterfacegoesdownthestateofentriesinthetablearechangedtoINACTIVE.ThisalsoappliesifaPCattachedtoaCiscophoneisshutdown,theentryimmediatelybecomesINACTIVE. Allentries(ACTIVEandINACTIVE)remainintheIPdevicetrackingtableuntilthemaximumtablelimitisreached(switchdependent),afterwhichtheoldestentrywillthenbereplaced. Thedefaultprobeinterval(agingtimer)is30seconds.Theidletimeoutisactuallytheprobeintervalmultipliedbythenumberofprobeentries(3),so90seconds(idletimer). Theswitchwillsendaprobeevery30seconds(agingtime)toeveryportthatithasanACTIVEentry. IftheswitchreceivesanARPpacket(request,replyorGARP)onaport,itresetstheagingtimebackto30secondsforthatport. IftheagingtimeexpirestheswitchsendsanARPprobetoverifythatthehostispresent.Ifthehostispresent,itsendsaresponsetotheswitchandtheswitchupdatesitsentry. IPDTprobesareintheformof: SenderIP:0.0.0.0        SenderMAC:switch_int_MAC TargetIP:attached_host_IP      TargetMAC:attached_host_ip Willseethebelowdebuglogmessageevery30secondsorwhenanARPpacketisreceivedfromthehost. Jan 9201712:21:19.063GMT:sw_host_track-notify:host_track_activate_entryNotifyotherfeatures:activate-(Gi1/0/10169.254.11.153(e8e0.b708.6c5e)VLAN:12ID:133ARP) ExampleofanIPmovingbetweenports: Jan 9201712:19:50.241GMT:%LINK-3-UPDOWN:InterfaceGigabitEthernet1/0/10,changedstatetoup Jan 9201712:19:51.143GMT:sw_host_track-obj_destroy:HostremovedfromAVL1HASH1INTF1(Gi1/0/210.17.10.81(e8e0.b708.6c5e)VLAN:12ID:133ARP) Jan 9201712:19:51.143GMT:sw_host_track-notify:host_track_delete_entry-Notifyotherfeatures:del-(Gi1/0/210.17.10.81(e8e0.b708.6c5e)VLAN:12ID:133ARP) Jan 9201712:19:51.143GMT:sw_host_track-obj_destroy:Hostentryfreed(Gi1/0/210.17.52.81(e8e0.b708.6c5e)VLAN:12ID:133ARP) Jan 9201712:19:51.143GMT:sw_host_track-obj_create:e8e0.b708.6c5e(169.254.11.153)NewmacinfoAVLnodecreated Jan 9201712:19:51.143GMT:sw_host_track-obj_create:e8e0.b708.6c5e(169.254.11.153)Cacheentrycreated Jan 9201712:19:51.143GMT:sw_host_track-notify:host_track_add_entry-Notifyotherfeatures:wiredadd-(Gi1/0/10169.254.11.153(e8e0.b708.6c5e) ExampleofthesameIPonsameport: Jan 9201712:45:31.540GMT:%LINK-3-UPDOWN:InterfaceGigabitEthernet1/0/11,changedstatetoup Jan 9201712:45:32.540GMT:%LINEPROTO-5-UPDOWN:LineprotocolonInterfaceGigabitEthernet1/0/11,changedstatetoup Jan 9201712:45:32.695GMT:sw_host_track-notify:host_track_activate_entryNotifyotherfeatures:activate-(Gi1/0/11169.254.11.153(e8e0.b708.6c5e) Cisco’srecommendationstotackletheduplicateIP0.0.0.0issuesare: Enterthenon-zerosourceIPinARPrequests.Thedownsideisthismakesitnon-RFCcompliant. DelaythefirstIPDTprobe(istriggeredbyalink-up)by10seconds. TounderstandwhythiswillresolvetheduplicateIPaddressissueweneedtofirstunderstandhowcollisiondetectionworks. CollisionDetection(RFC5227) CollisiondetectionisusedbytheDHCPclienttoensurethattheIPithasbeenassignedisn’talreadyinuse.ItdoesthisbysendingARPrequestsourcedfrom0.0.0.0withthetargetbeingtheDHCPassignedIPaddressitwantstouse. SenderIP:0.0.0.0      SenderMAC:PC_MAC TargetIP:PC_DHCP_IP    TargetMAC:00:00:00:00:00:00 FromthetestsIperformeditseemsthatWindowsdoescomplywiththeRFCintermsofhowitperformscollisiondetection.Asummarizationof theprocessforcollisiondetectionaspertheRFC: HostwaitsforarandomtimeintervalselecteduniformlyintherangezerotoPROBE_WAIT(1)secs. ItwillsendPROBE_NUM(3)probepacketsintotal Eachoftheseprobepacketsisspacedrandomlyanduniformly,PROBE_MIN(1)toPROBE_MAX(2)secsapart(tostopsendinginitialprobessimultaneouslyiflotsofhostspoweredonatsametime). Ifduringthisperiod,fromthebeginningoftheprobingprocessuntilANNOUNCE_WAIT(2)secondsafterthelastprobepacketissentoneoftwoconditions(seebelow)ismetthehostMUSTtreatthisaddressasbeinginusebysomeotherhost. IfclassedasfreehostMUSTannouncethatitiscommencingtousethisaddressbybroadcastingANNOUNCE_NUMARPannouncements(GARP),spacedANNOUNCE_INTERVAL(2)secondsapart.Thehostmaybegin legitimatelyusingtheIPimmediatelyaftersendingthefirstofthetwoARPannouncements. Thereforetheclientcanwaitupto9secondswhenmonitoringforacollision: Minimumtime=6 1(PROBE_WAIT)+[3(PROBE_NUM)x1(PROBE_MIN)]+2(ANNOUNCE_WAIT) Maximumtime=9  1(PROBE_WAIT)+[3(PROBE_NUM)x2(PROBE_MAX)]+2(ANNOUNCE_WAIT) The2possibleconditions(reasons)foritdetectingacollisionare: (a)LooksasifanotherhostalreadyhasandisusingtheIPaddress.Ifduringdetectionperiodthehostreceives: AnyARPpacket(Request*or*Reply)ontheinterfacewheretheprobeisbeingperformed Wherethepacket’s‘senderIPaddress’istheaddressbeingprobedfor ThenthehostMUSTtreatthisaddressasbeinginusebysomeotherhost (b)AnotherhosthasbeenassignedtheIPatthesametimeorIPDTprobe.Ifduringdetectionperiodthehostreceives: AnyARPProbewherethepacket’s‘targetIPaddress’istheaddressbeingprobedfor Andthepacket’s‘senderhardwareaddress’isnottheMACofanyofthehost’sinterfaces ThenthehostSHOULDsimilarlytreatthisasanaddressconflictandsignalanerrorto theconfiguringagentasabove HowdoesthisrelatetoIPDT IntermsoftheduplicateIPissueseenwhenusingIPDTitisdowntoitmatchingconditionb. IftheCiscoIPDTprobehappensduringthis9secondCollisiondetectionwindow,asbothprobeshaveasenderIPof0.0.0.0andthe sametargetIPthewindowsclientmarkstheIPasusedandwillwaits30secondsbeforetryingagainforanotherIP. TheDHCPserverwillmarkthatIPasabadaddressmeaningitcannolongerbeused. Therefore,ifyougobacktoCisco’sworkarounds: Enterthenon-zerosourceIPinARPrequests(makesitnon-RFCcompliant) SolvestheproblemasthesourceIPaddressesarenowdifferentmeaningtheclientwont seeitasanotherhostdoingcollisiondetectionforthesameassignedDHCPIP(targetaddress) DelaythefirstIPDTprobe(istriggeredbyalink-up)by10seconds Asthemaxtimeis9seconds,thisdelayof10secondsgivestheclienttimetogothroughitscollisiondetectionprocessandavoidtheconflict WhenaWindowsclientbootsupitdoescollisionforitsAPIPAaddressstraightawaywhichwillcausetheagingtimetoresetsogiveyou30seconds(unlessanotherisreceivedARPinbetween)beforetheIPDTwillattempttoprobeagain.Addingthedelaytothisthatgives40secondsforthePCtobootupandgetanIPaddress. IfPCsdoaPXEbootatstartupthe10seconddelaymaynotfixtheissueasitisdependentonhowlongyoursystemtakestostartup.PXEwillnotdoacollisiondetectionbutthiswillcausetheIPDTentrytogoACTIVE,meaningaprobewillbesent30secondsafterwardsasisunlikelyanyotherARPswillbeheardbytheswitchonthatport(toresettheIPDTtimer)whilstthePCisbooting.Thereforeifittakes35secondsforyourPC tobootupthe10seconddelaywillnotfixtheproblem. BecauseofPXEbootandPCboottimesipdevicetrackingprobedelay20didnotfixtheissueforus.Insteadweusedthefollowinganditresolvedtheproblem. Settingthecountto2ensurethatitstillhadthedefaultbehaviorof90secondsbeforeentriesbecomeINACTIVE. ipdevicetrackingprobeinterval45 ipdevicetrackingprobecount2 Someusefulcmdforseeinghowitworks,butbewarnedwillgeneratealotoflogsifyouhavemanyIPDTentries. debugipdevicetrackingerror debugipdevicetrackingnotify debugipdevicetrackingobj-create debugipdevicetrackingobj-destroy ForreferencebelowismyWindowstestmachinesbootprocessandtimes. 1.0second-PXEasksforIPusingDHCPREQUEST 2.IsPXEassignedanIPbyDHCP 3.RequestsTFTPdata 4.26seconds-PCsendsARPrequest(broadcast) SenderIP:0.0.0.0MAC:PC_interface_MAC TargetIP:169.254.x.xMAC:00:00:00:00:00:00 5.40seconds-SendsDHCPDISCOVER/REQUEST 6.DHCPACK 7.PCsends3requests,1everysecond(Windowscollisiondetection) SenderIP:0.0.0.0MAC:PC_interface_MAC TargetIP:PC_DHCP_IPMAC:00:00:00:00:00:00 8.44seconds-PCsendsaGARP SenderIP:PC_DHCP_IPMAC:PC_interface_MAC TargetIP:PC_DHCP_IPMAC:00:00:00:00:00:00 9.SendsseveralAPRsfordefaultgateway.AstheswitchreceivesthisitwontsendanyprobesuntilnoARPreceivedformorethan30seconds 10.68seconds-ClientsendsaDHCPInformtosay“IamusingthisIP”,whichwillgetACKfromDHCPservers Next Post→ RELATEDPOSTS SD-WANtransport-sideBGP 27September2022 BGPMPLS-VPNOptionB 1May2022 SD-WANservice-sideMPLSVPN 22February2022 VirtualTunnelInterface(VTI)VPN 27July2021 AutomateLeafandSpineDeployment-Part6 23March2021 RECENTPOSTS SD-WANtransport-sideBGP 27September2022 BGPMPLS-VPNOptionB 1May2022 PynetboxAPIcallsusingvariables 2March2022 SD-WANservice-sideMPLSVPN 22February2022 F5BIG-IPDNSExpress 9January2022 FEATUREDTAGS ansible checkpoint cisco eve-ng firewall napalm nornir python routing sd-wan switching vmware vpn