Jump to content

Recommended Posts

Hey all, 

For those of you who dont know...  I am the Product Manager for the newly introduced Network Probe Gen 2.

After seeing many, and quite often the same, questions being asked since Network Probe Gen2 launched last week, I felt that now is a good time for me to create an FAQ for all things related to Gen2.

This is not intended to replace, or even be an alternative to, the Official Docs (Click here to read those!), but more so something to supplement the information that is already out there. If, after reading this, you still have some questions - or would like further detail on any existing content, then PLEASE reach out to me on Slack (@martynkeigher), or just reply to this post. After I answer each new question - I'll move them into the FAQ, so they don't get lost.

So... lets see how this goes.... 

__________________________________

Q1: I’ve just update to [Automate 12] Patch 10... am I on the latest and greatest now?

Yes and No. While you are on the version that introduced Network Probe Gen 2, you must also ensure that the ‘ConnectWise Network Detection’ solution is kept up-to-date, via the Solution Center.

 

Q2: I’ve just updated to [Automate 12] Patch 10, updated to Gen2, and I cannot see my ESX hosts (For Virtualization Manager)!  Why is that?

This was a known-issue during pilot and was fixed in v1.1.1.6 of LTNet.dll, within the ‘ConnectWise Network Detection’ solution. Please ensure you have (at least) that version, and then restart the remote (probe) agents to force the update of the new dependency. (See Q10 for more info on this.)

 

 Q3: Can I update all my probes to Gen2 with an ‘easy button’?

No. There is currently no feature, or future-plan, to mass-update all your probes Gen2.

 

Q4: So… with the new probe, I can upgrade as I go? It doesn't upgrade all my probes when I update to Patch 10, does it?

Installing Patch 10 will not auto-upgrade your Gen1 probes to Gen2. You can upgrade your probes at your own pace.

 

Q5: As soon as I enable a Gen2 probe, a discovery scan immediately starts. I’d rather configure some settings first, and then manually start the discovery scan.  How would I do that?

Network Probe defaults was also introduced in Patch 10. They can be found under System > Configuration > Network Probe. If you do not want you probes to immediately scan, adjust the default scan windows to something like… 10pm (start time) & 3am (end time), or alternatively, set the Scan Frequency to On-Demand.

 

Q6: I’m in the middle of converting all my Gen2 probes… How can I keep track of my Gen1 and Gen2 probes?

There is currently a data view in development for this. It will be added to the ‘ConnectWise Network Detection’ solution once available. Until then though, you could use this: (Disclaimer: I accept no responsibility if you 'get creative' with this SQL statement and, by doing so, you break stuff!)
 

SELECT cl.Name AS 'Client',
	l.name AS 'Location',
	CONCAT (c.name,'(',c.computerid,')') AS 'Probe Agent (ID)',
	IF(c.LastContact < DATE_ADD(NOW(),INTERVAL 60 SECOND),'Online','Offline') AS 'Status',
	IF(c.locationid IN (SELECT locationid FROM probeconfiguration),'Generation 2','Generation 1') AS Gen
FROM clients cl
JOIN computers c USING (clientid)
JOIN locations l USING (locationid)
WHERE c.computerid IN (SELECT probeid FROM locations)
ORDER BY cl.name,l.name

 

Q7: I just built a bunch of new Detection/Collection templates. Are they still honored?

Yes. They will still be honored.

 

Q8: I don’t like Gen2 and all its new, cool awesomeness!! Can I roll back to (or re-install) a Gen 1 probe?

No. Assuming you are on at least Patch 10, once you have promoted a Gen1 probe to Gen2, or just installed a Gen2 probe, there is no rolling back to Gen1.

 

Q9: So… What’s this I hear about a new discovery method(s)? It’s been brought up, but never actually broken down & explained. Explain it for me? 

The Gen2 ‘Discovery Scan’ uses 3 methods to discover a device. After the initial ping-sweep of the subnet(s) is complete…

  1. The 1st check is NETBIOS. This checks to see if port 139 is open. This gives us a ‘Missing Device’ count.
  2. The 2nd check is a SNMP check against the device to determine its MAC address.
  3. The 3rd (and optional) check is the ARP Table scanning. This is not enabled out of the box but can be enabled via the Network Probe Setting, ‘Enable MAC Address Scanning’.

 NOTE: Just like with the old Gen1 probe… A MAC Address is required to uniquely identify the device - if the probe cannot determine a MAC address, then it will not create a Network Device.

 

Q10: When you say “SNMP Check” … what do you mean?

If we can communicate with a device using SNMP (community string) we walk the device’s MIB for 2 specific OIDs (Object Identifiers) to obtain the MAC. If this check does not return a MAC address, then we will not create a Network Device entry for it.

 

OID Check

Example

Returns

Check 1

1.3.6.1.2.1.4.20.1.2.<Device IP Address>

1.3.6.1.2.1.4.20.1.2.10.10.83.1

<InterfaceID> 

Check 2

1.3.6.1.2.1.2.2.1.6.<InterfaceID>

1.3.6.1.2.1.2.2.1.6.27

MAC Address

I'll illustrate.... 

  • Check 1 (returns the Interface ID, in this case 27)

image.png.2a22e427229b63be5cf4487a9ec193dd.png

  • Check 2 (returns my MAC address - and the one that will be used in the networkdevices database table).

image.png.c116b6f3f0cdca71424e46352f90d758.png

 

FYI: The VMware MIB stores these values in different OID's for ESX hosts:

Check 1: InterfaceID:        1.3.6.1.2.1.4.34.1.3.1.4.<IPAddress>

Check 2: MAC Address:    1.3.6.1.2.1.2.2.1.6.<InterfaceID>

 

Q11: I heard something about the Network Probe being more focused around the location… Can you elaborate on that?

During pilot, we re-worked how the probe is associated to the location, and now the location has more of a role to play for Network Probes. This was a huge undertaking, and ‘laid the groundwork’ for implementing many other enhancements already on the roadmap. A good use-case that this solves for today is when a probe agent stops functioning, and cannot be brought back online, e.g.: a BSOD. Should a probe agent become unrecoverable, then you can now elect a different agent to be a probe at the same location, as the settings are now tied to the location and not the agent itself, so the new probe will ‘pick up where the old one left off’.

 

Q12: We haven't used the network probe much until this release and it's bringing up some questions now. So, should we be enabling SNMP on almost all networking devices?

For best results with Gen2 you must have SNMP enabled on SNMP compatible devices.  Doing that will set you up for success. No doubt about it!! From Automate’s perspective… You can’t really say you are ‘managing devices’ unless you have SNMP connectivity to them!

 

Q13: The new network map is cool, but it doesn't show nearly everything the probe sees. Is that normal?

Network Probe Gen2 places a heavy dependency on SNMP for device discovery. If we cannot get the MAC address of a device, then we cannot add it as a Network Device. Gen2 was designed to only ‘bring in’ devices that it can manage. Be sure that you have SNMP enabled, on your SNMP compatible devices, and can validate the MAC address (as referenced earlier). If you can, then please contact support, and be sure to have the following 'qualifiers':

  • On Automate 12 Patch 10 (build 451), minimum.
  • On at least v1.1.1.6 of LTNet, (distributed in 'ConnectWise Network Detection' via Solution Center).
  • Communication to the device, via SNMP, is returning the MAC Address.

 

Q14: SNMPv3… yes? no?

SNMPv3 is still an option with the new Network Probe. You can configure that at a probe level. We recommend that you set that if all your devices, that are configured for SNMPv3, are using the same authentication/encryption passwords and methods.

 

Q15: I'm seeing more devices in my network device list and my computers list then I am seeing on my network map. Any idea what I may be doing wrong?

Only devices discovered via the Gen2 probe have the potential to be placed on a map. Devices discovered during Gen1 will not show. In addition to this, (due to how we still rely on the MAC address of a device to give it a DeviceID), you may also find that while you can see network devices in your list, they may not show on the map, as they are Gen1 discovered devices. (The MAC already existed in the DB table, so the device can’t be ‘re-added’). Extreme scenarios where this has been witnessed is where, after a discovery scan shows multiple devices yet only ONE device is on the map. This also leads into the next question…

 

Q16: Thoughts/process on… ‘Nuking’ my current Gen1 probes and configs (never really got into it) to setup the new Gen2 stuff? Any ‘best practices’ around this?

Given the feedback we received during pilot and the new probe being viewed as a time to ‘start-over’ and ‘re-do probes’ for most Automate admins, the best practice approach to this would be to disable the probe role from the agent and then re-enable the probe as a Gen2. The commands that proved to be most popular on existing probes (after adjusting Network Probe defaults) was:

  • Purge Network Devices
  • Disable Gen1 Probe
  • Enable (Gen2) Probe
  • Start Discovery Scan

 

Q17: I disabled the network probe on this agent but now it's saying I can't retire it, because the network probe is enabled!? Why is that?

The ‘Disable Probe’ command is a database focused command, so the Control Center may not acknowledge the change until you ‘Reload System Cache’ & then refresh the Computers view.

__________________________________

Extra Info:

Log files:

  • C:\Windows\LTSVC\lt_LTNetAccess.log
  • C:\Windows\LTSVC\lt_LTNetMap.log
  • C:\Windows\LTSVC\lt_LTNetScan.log
  • C:\Windows\LTSVC\MyNetwork.xml

 

Support Tools: (attached to this post!)

 

Database Tables:

  • Gen1
    • probeconfig
  • Gen2
    • probeconfiguration (& many other probe**** tables).
    • networkswitchmap
      • No point editing as table redraws after each Discovery scan
      • SwithMac is the Parent (uplink) device
      • DeviceMac is the child device
      • HH-HH-HH-HH-HH-HH = Top level device
      • GG-GG-GG-GG-GG-GG = Internet (WAN) connection

 

Official Links:

 

</END> 

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

Hi Martyn,

Do you have any updated documentation on how to use the new probe? We've upgraded the probes on in our Automate, however, now when I try to run network probe commands (Scans or re-detects) nothing seems to happen. The Probe Commands windows doesn't show anything either. I used to be able to force this with the old one. Has that changed?

Share this post


Link to post
Share on other sites

Is anyone else having issues with the new probe not picking up vsphere servers?

We have deployed at 2 clients purged devices and redetect and it seems like it skips them.

I can see in the plugin_vm log where it is trying to use the vsphere credentials against devices but not the actual vsphere servers it needs to detect.

It seems to be skipping them and they don't show up anywhere.  We can ping them from the probe machine.

Got a ticket open with support.  They said we had to enable SNMP on the vsphere host.  I am not 100% sure this needs to be done but we did anyway.  Still did not work.  I can SNMP walk the device from the probe server and get to the ESX OID's listed from above.

Anyone else seeing this or have any ideas?  (Still working fine with the old probe.)

Let me know.

Thanks

 

Edited by rolodex

Share this post


Link to post
Share on other sites

I can confirm the behaviors @avdlaan describes: scans/re-detections don't actually seem to be running when I attempt to trigger them manually, and my Probe Commands window remains empty even after attempting to initiate a scan or re-detection.

Following the steps outlined here for upgrading from Gen1 to Gen2, I purged network devices from a probe for a selected client, then deactivated the Gen1 probe. When I re-activated and went through the Gen2 wizard, the probe functionality returned, but none of the options to perform manual actions appear to be working, and the Probe Commands window is empty. I have tried initiating a rescan via the "Play" button on the Device Discovery pane in the Overview screen and also tried via Commands > Probe > Initiate Network Scan and Commands > Probe > Run Device Detection. I also ensured the probe scan settings were set to have Only run discovery in scan window = UNCHECKED.

I want to make sure I'm not missing something regarding manually initiating scans/re-detections, as well as the Probe Commands window appearing blank.

Finally, I am interested in finding out what benefit we gain from activating the MAC Address Scanning feature? I have seen the documentation reviewing how it fits into discovery as the optional third step. But I haven't been able to determine what that third step of scanning the MAC addresses actually accomplishes. Is it enabling some new functionality? Does it allow more reliable agent push to detected devices?

We face multiple client probes (Gen1 and Gen2) in which devices are mis-detected (Windows desktops appearing as "Manufacturer = Roku" or "Manufacturer = Apple, Inc.") or simply not allowing agent install (multiple VM servers at one client, for instance). If enabling MAC address scanning will remedy one or more of these situations, that's a strong argument for turning it on. If it is intended to perform some other function ... what might that be?

Share this post


Link to post
Share on other sites
On 12/6/2018 at 3:31 AM, rolodex said:

Is anyone else having issues with the new probe not picking up vsphere servers?

We have deployed at 2 clients purged devices and redetect and it seems like it skips them.

I can see in the plugin_vm log where it is trying to use the vsphere credentials against devices but not the actual vsphere servers it needs to detect.

It seems to be skipping them and they don't show up anywhere.  We can ping them from the probe machine.

Got a ticket open with support.  They said we had to enable SNMP on the vsphere host.  I am not 100% sure this needs to be done but we did anyway.  Still did not work.  I can SNMP walk the device from the probe server and get to the ESX OID's listed from above.

Anyone else seeing this or have any ideas?  (Still working fine with the old probe.)

Let me know.

Thanks

 

Yeah mate, we are having the same problem. Lenovo (IBM) servers at the moment. I think they are both 6.7 if I recall what my guys said. No fix I am aware of.

Share this post


Link to post
Share on other sites

I finally got support lined up to talk to today.  The servers I am trying to pick up are 6.5 u1 and 6.5 u2.

Will update when I know more. 

Thanks

Share this post


Link to post
Share on other sites

Talked with support.  Shame on me for not trying - but they had me check the box on the probe computer.

On the network probe, settings, Scan tab.  Check the Enable MAC scanning. (What JosefNT mentioned above) I did this, restarted the agent, initiated a scan and they are now being detected. 

By turning this on it basically turns on the old probe 1 scanning functionality.  It uses the MAC address table from the ARP cache on the probe device for building the network device list.

I was told the reasoning for not having it on was the old probe picked up things you could not manage.  Their take was if you can't manage it why have it in there.  I am looking at it a little differently.  I want to see it even if I can't manage it so I don't miss something. 

I am leaning toward just checking the box and being done vs having to enable SNMP on all my servers.  That would be a ton of work for my team for no benefit.

If someone has a different take or thinks I am missing something, let me know.

Thanks

 

 

Share this post


Link to post
Share on other sites

Glad it worked for you. Unfortunately that doesn't work if the probe is not in the same subnet as the esxi server, so for us, no dice. I have decided to turn Mac address scanning on everywhere, because like you, I don't want an automate distorted view of what's there, I want what is really there. 

Share this post


Link to post
Share on other sites

I can see why the MAC thing would not work for remote subnets.

I forgot to add - the tech told me that Patch 12 has some fixes for the new probe - we are implementing that this weekend.  He also said they were having issues where if you did not have SNMP enabled on your switches - it could possible cause issues with detection as well.  sounds strange but that is what they are telling me.  Hopefully the new patch fixes the remote stuff but not sure.

Share this post


Link to post
Share on other sites

I asked support but figured I'd post here also.  In the Gen 2 probe it looks like there is only one checkbox, Enable Remote Agent Deployment. The docs at https://docs.connectwise.com/ConnectWise_Automate/ConnectWise_Automate_Documentation/070/175/120/030 say it will try to install the agent when a PC is next scanned.

Is there any way to allow (only) the manual push via password  (commands/probe/deploy Automate agents) or has that essentially been removed in favor of automatic push?

We have typically either manually pushed, or set up a domain login script to install the agent if we want it automatic.  That way it isn't trying to install on any other devices on the network, third party PCs, etc.

 

Edit: Disregard my question.  I was pointed to Deployment Manager.  I had seen Deployment Manager a while back and then totally forgot about it since it essentially duplicated the probe feature.  Interesting it still refers to its settings as “probe” which is apparently not related to the gen 2 probe.

Edited by SteveYates

Share this post


Link to post
Share on other sites

Anyone having the problem where the deployment manager says "Device failed deployment readiness checks"?

I've got the password set at:

Location
Probe
Deployment Manager
LTService

Password doesn't have any special characters either. 

 

Not sure if anyone has run into this issue before? Seems to be happening accross all our gen2 probes.

 

 

  • Like 3

Share this post


Link to post
Share on other sites

Hi Martyn,

Big fan of the Gen2 probe. So good work.

I've got one question about monitoring snmp or Collecting SNMP info using emplates. Hope you can give me an example so I can use it in my search to knowledge because there's no info on the documentation site from Automate.

There's a firewall. Lets call it a WatchGuard firewall. 😉
A Walk on the firewall shows me a ton of information but the information has a lot of variables. So there's no static OID to monitor.

There's a Index table showing me indexing numbers for the next tables.
The next table is the name table. Relating to the index above.
The next table is the Status. Also relating on the index above.

Is there a way to do this: Look for the name in the Name table. Find the index related to this name, find the status related to the index.

Let me show:

Capture.thumb.JPG.0bde0d9a90048da0c624631e5a4610f5.JPG

 

1 find oid with value IKED
2 Find related index
3 Find status from this value with the related index.

I guess this should be able with the SNMP Collection Templates and Sub Procedures. However..... how?

Hope you can point me in the right direction.

 

Kind regards,

Bas van der Zanden

Share this post


Link to post
Share on other sites
On 2/19/2019 at 11:55 PM, zubz said:

Anyone having the problem where the deployment manager says "Device failed deployment readiness checks"?

I've got the password set at:

Location
Probe
Deployment Manager
LTService

Password doesn't have any special characters either. 

 

Not sure if anyone has run into this issue before? Seems to be happening accross all our gen2 probes.

 

 

We are seeing exactly the same thing!

  • Like 1

Share this post


Link to post
Share on other sites
Posted (edited)

Moved to full thread

Edited by syscal
Not sure why I posted it here...

Share this post


Link to post
Share on other sites

Seems to be a hit and miss with Gen 2 Probe picking up Esxi.

I have tried Mac address scanning, Disabling Ilo IP address scan, some Esxi hosts are picked up with their correct IP and Mac address while other sites the probe picks it up but uses the same IP as the Ilo. On other sites the ESxi isn't picked up at all.

SNMP is enabled! Auvik is authenticating with SNMP.

I've been reading and patiently waiting to get this resolved, not sure what else to do, any help appreciated.

 

Share this post


Link to post
Share on other sites
53 minutes ago, Namik said:

Seems to be a hit and miss with Gen 2 Probe picking up Esxi.

I have tried Mac address scanning, Disabling Ilo IP address scan, some Esxi hosts are picked up with their correct IP and Mac address while other sites the probe picks it up but uses the same IP as the Ilo. On other sites the ESxi isn't picked up at all.

SNMP is enabled! Auvik is authenticating with SNMP.

I've been reading and patiently waiting to get this resolved, not sure what else to do, any help appreciated.

 

Hi Namik,

Can you verify if your probe is on another network segment as the Esxi hosts are? I discovered something strange in this setup and reported it to CW support. It's been escalated to DEV in the meantime. There's the same with iLO5 and WatchGuard firewalls. Let me sketch the situation:

"Got a probe in network 192.168.1.0/24. The probe has just one network interface. Traffic to another network is being arranged by a routing switch or firewall/router to the segment 192.168.2.0/24. The probe is scanning both subnets. It has no problem with 192.168.1.0/24 but in the other network it does not find all devices. All ports are open."

I found it has to do with the detection method from Gen2 Probes. It's being explained in Q10 from this topic. When manually discovering a... let say HP iLO5 in this case, it finds the IP address. So it starts to find the index table from the device. However, the first available index refers to 1.3.6.1.2.1.4.20.1.2.127.0.0.1. So the probe is confused... is it me? not sure.. what to do? ah just leave it.

Now, when you assign another NIC to the probe and add network segment 192.168.2.0/24 to it, the probe seems to be smart enough to use that one to scan instead of routing the traffic. But when you scan the device now, the first available index refers to 1.3.6.1.2.1.4.20.1.2.192.168.2.30. Now it know what to scan and the device will show up in the network device list.

This is a known issue and logged with 11576922. (https://university.connectwise.com/university/reports/automateknownissues.aspx)
My originating ticket was 11737449 and was merged into this one. I've never mentioned a Esxi host but only WatchGuard firewalls and iLO5 devices. But they told me it's the same problem. It seems like a fix is being written because the status is "review by PM".

Share this post


Link to post
Share on other sites

Quick Comment: MAC Scanning will find devices by ARP responses, so it can only ever work when in the same VLAN. For cases where you are not finding things on another VLAN behind a Layer 3 device, MAC scanning does nothing. (And adding a second NIC connected to the second network allows MAC scanning to work as you are now in the same VLAN).

21 hours ago, Duvak said:

...
I found it has to do with the detection method from Gen2 Probes. It's being explained in Q10 from this topic. When manually discovering a... let say HP iLO5 in this case, it finds the IP address. So it starts to find the index table from the device. However, the first available index refers to 1.3.6.1.2.1.4.20.1.2.127.0.0.1. So the probe is confused... is it me? not sure.. what to do? ah just leave it.

Now, when you assign another NIC to the probe and add network segment 192.168.2.0/24 to it, the probe seems to be smart enough to use that one to scan instead of routing the traffic. But when you scan the device now, the first available index refers to 1.3.6.1.2.1.4.20.1.2.192.168.2.30. Now it know what to scan and the device will show up in the network device list.

Are you saying that if you SNMPwalk the device from another subnet, the first index in the interface table will refer to 1.3.6.1.2.1.4.20.1.2.127.0.0.1, but if you walk from an IP in the 192.168.2.x network the same index will point to 1.3.6.1.2.1.4.20.1.2.192.168.2.30? That would mean the device is acting differently based on you IP. My suspicion is that that is not the case. I expect it is getting the same results in the same order as before, but if MAC scanning is enabled and you are connected directly it doesn't need the SNMP data and can successfully detect the device by the ARP response.

Share this post


Link to post
Share on other sites
On 5/24/2019 at 3:31 PM, Duvak said:

Hi Namik,

Can you verify if your probe is on another network segment as the Esxi hosts are? I discovered something strange in this setup and reported it to CW support. It's been escalated to DEV in the meantime. There's the same with iLO5 and WatchGuard firewalls. Let me sketch the situation:

"Got a probe in network 192.168.1.0/24. The probe has just one network interface. Traffic to another network is being arranged by a routing switch or firewall/router to the segment 192.168.2.0/24. The probe is scanning both subnets. It has no problem with 192.168.1.0/24 but in the other network it does not find all devices. All ports are open."

I found it has to do with the detection method from Gen2 Probes. It's being explained in Q10 from this topic. When manually discovering a... let say HP iLO5 in this case, it finds the IP address. So it starts to find the index table from the device. However, the first available index refers to 1.3.6.1.2.1.4.20.1.2.127.0.0.1. So the probe is confused... is it me? not sure.. what to do? ah just leave it.

Now, when you assign another NIC to the probe and add network segment 192.168.2.0/24 to it, the probe seems to be smart enough to use that one to scan instead of routing the traffic. But when you scan the device now, the first available index refers to 1.3.6.1.2.1.4.20.1.2.192.168.2.30. Now it know what to scan and the device will show up in the network device list.

This is a known issue and logged with 11576922. (https://university.connectwise.com/university/reports/automateknownissues.aspx)
My originating ticket was 11737449 and was merged into this one. I've never mentioned a Esxi host but only WatchGuard firewalls and iLO5 devices. But they told me it's the same problem. It seems like a fix is being written because the status is "review by PM".

Hi Duvak,

Thanks for the input. I had read about diff subnets, however only one network segment. Whats even more frustrating is that I have setup several clients at different locations with same HP servers, same modem, routers, same EsXI ver 6.5 HP Customised, using same single network config at all locations. Two sites detected ESXi correctly after many attempts at mac scanning and disabling scanning of ilo IP. The other sites either don't pick it up at all or pick it up using the same IP as ilo but showing diff mac. see pic. Cheers

LTClient_Saklu3ckcY.png.ea31a16e836df6a716c0cd30f266ef05.png

Share this post


Link to post
Share on other sites
On 5/25/2019 at 4:59 AM, DarrenWhite99 said:

Quick Comment: MAC Scanning will find devices by ARP responses, so it can only ever work when in the same VLAN. For cases where you are not finding things on another VLAN behind a Layer 3 device, MAC scanning does nothing. (And adding a second NIC connected to the second network allows MAC scanning to work as you are now in the same VLAN).

Are you saying that if you SNMPwalk the device from another subnet, the first index in the interface table will refer to 1.3.6.1.2.1.4.20.1.2.127.0.0.1, but if you walk from an IP in the 192.168.2.x network the same index will point to 1.3.6.1.2.1.4.20.1.2.192.168.2.30? That would mean the device is acting differently based on you IP. My suspicion is that that is not the case. I expect it is getting the same results in the same order as before, but if MAC scanning is enabled and you are connected directly it doesn't need the SNMP data and can successfully detect the device by the ARP response.

Hi Darren,

The problem is with both MAC disabled and Enabled. Because when I do a arp -a on the probe, it does not show the IP address from the iLO of Firewall. Even when I visit the site or ping it, it will not appear in the ARP table. So following the scan method: step 1: netbios. All ports are open but no show. 

And Step 2 (SNMP Check):
Sorry, i've been not very clear. Let me show you. This is a SNMP Walk to a firewall device scanning from another subnet:

image.png.9d286bbcb98e5ebb5e945dc4d31d1a25.png

You see index 1 is bound to 127.0.0.1. 

Scanning from the same probe a Switch for example, it shows me this:

image.png.5924340ae33ffb180671b69fe7bd9978.png

That's what we want to see. 

So we've got a visual on whats going wrong in the detection according to step 2 in Q10 in this post. So the alternative would be Step 3, Arp scan. However, the iLO device/watchguard firewall device does not show up in the ARP table. Even when i Ping it of visit the website.i'm not that familiar with ARP so I guess this is expected behavior when a device is in another subnet?

So you see: even when all requirements are met, the device will not show up in the networkdevice list.

image.png

Share this post


Link to post
Share on other sites

ARP only works in the same VLAN. Normally it only works in the same subnet. When you ping a device in your subnet, the OS will broadcast an ARP request and the device MUST respond and will show in the arp table even if it is completely firewalled and responds to no services. When the device is in a different subnet, the OS doesn't bother with ARP and will only send the packet to the gateway. MAC Scanning works across subnets but only in the same VLAN, specifically it ignores if the device is in the same subnet and broadcasts the ARP request anyways. If the device is in the same VLAN it will still respond, if it isn't, it won't. (And if you have a device that performs PROXY-ARP, things get screwed up so you must disable PROXY-ARP if you use MAC Scanning).

Getting different SNMP WALK results from different locations is super weird, unless: 
A) You are doing the scans at different enough times that the cache data is changing completely. (You would need to test from both locations at the same time to rule that out).
B) You are actually getting two different devices. (If you are scanning from a remote subnet, you are reaching whatever device the gateway has learned for that IP. In the same subnet, your computer is getting it's own ARP response. Two devices responding to the same IP can cause this. Check the ARP cache from the gateway, and the ARP cache from the machine in the local subnet, and confirm that both are targeting the same MAC for the same IP)
C) The SNMP Implementation on the device is screwy. (Check for view filtering, where the local subnet is allowed one level of access or OIDs and another subnet is allowed something different). But all bets are off the if device is giving bad SNMP results. A firmware update would be the only option. I have seen broken SNMP be resolved with a firmware update.

Learning IP/MAC/ARP information is actually a function provided by the gateway device. Specifically, if the gateway for the subnet is learned and is responding to SNMP, it will be probed to learn the ARP cache. The information gleaned from there will be used to learn the IP and MAC for devices.  You can see this by making your probe scan only 1 small subnet (say a /30) that includes the layer 3 device for multiple subnets. It will learn the IP/MAC for that device, and will also learn other devices that are not part of the scan. It will learn every device that it can based on the data returned from the gateway. Even though you have only specified 4 IP's to be scanned, you will see that many more are learned. It isn't strictly necessary that a device respond to SNMP itself if a gateway device can provide the information needed. (But if a device CAN respond to SNMP, it is able to identify itself even if the gateway doesn't)

Not sure how this specifically helps your situation, as I am saying that the probe should be able to find the device in many ways and you are experiencing it failing on all levels to find it.... But maybe knowing a little more about the operations behind the scenes will help you figure out how you can help it find the device in your case.

 

Share this post


Link to post
Share on other sites
On 4/29/2019 at 3:23 PM, Ben Smith said:

We are seeing exactly the same thing!

Also same here.  2019.5

Have an open case, and its been escalated.  This thing makes me want to cry.... :)

Share this post


Link to post
Share on other sites

Miracle of Miracles!!

This morning I checked and sure enough, ESXI is found!!

But iLO is missing.

Besides disabling scanning the ilo IP address and enabling MAC scanning, I didn't do anything else.

I have no clue why it finds it sometimes and not at other times. If it's first come first found then that's stupid!

Share this post


Link to post
Share on other sites
12 hours ago, DarrenWhite99 said:

ARP only works in the same VLAN. Normally it only works in the same subnet. When you ping a device in your subnet, the OS will broadcast an ARP request and the device MUST respond and will show in the arp table even if it is completely firewalled and responds to no services. When the device is in a different subnet, the OS doesn't bother with ARP and will only send the packet to the gateway. MAC Scanning works across subnets but only in the same VLAN, specifically it ignores if the device is in the same subnet and broadcasts the ARP request anyways. If the device is in the same VLAN it will still respond, if it isn't, it won't. (And if you have a device that performs PROXY-ARP, things get screwed up so you must disable PROXY-ARP if you use MAC Scanning).

Getting different SNMP WALK results from different locations is super weird, unless: 
A) You are doing the scans at different enough times that the cache data is changing completely. (You would need to test from both locations at the same time to rule that out).
B) You are actually getting two different devices. (If you are scanning from a remote subnet, you are reaching whatever device the gateway has learned for that IP. In the same subnet, your computer is getting it's own ARP response. Two devices responding to the same IP can cause this. Check the ARP cache from the gateway, and the ARP cache from the machine in the local subnet, and confirm that both are targeting the same MAC for the same IP)
C) The SNMP Implementation on the device is screwy. (Check for view filtering, where the local subnet is allowed one level of access or OIDs and another subnet is allowed something different). But all bets are off the if device is giving bad SNMP results. A firmware update would be the only option. I have seen broken SNMP be resolved with a firmware update.

Learning IP/MAC/ARP information is actually a function provided by the gateway device. Specifically, if the gateway for the subnet is learned and is responding to SNMP, it will be probed to learn the ARP cache. The information gleaned from there will be used to learn the IP and MAC for devices.  You can see this by making your probe scan only 1 small subnet (say a /30) that includes the layer 3 device for multiple subnets. It will learn the IP/MAC for that device, and will also learn other devices that are not part of the scan. It will learn every device that it can based on the data returned from the gateway. Even though you have only specified 4 IP's to be scanned, you will see that many more are learned. It isn't strictly necessary that a device respond to SNMP itself if a gateway device can provide the information needed. (But if a device CAN respond to SNMP, it is able to identify itself even if the gateway doesn't)

Not sure how this specifically helps your situation, as I am saying that the probe should be able to find the device in many ways and you are experiencing it failing on all levels to find it.... But maybe knowing a little more about the operations behind the scenes will help you figure out how you can help it find the device in your case.

 

Hi Darren,

Thanks for taking the time to explain the principles of the probe system. This made me think a bit more out-of-the box. Yes, the situation is not only a different subnet, it's a different VLAN. So thats why ARP doesn't show the devices. So next is de SNMP scan. My probe is configured to not scan it's own gateway (which is the firewall) because the management IP for the firewall is in a different VLAN. So i've added the gateway IP to the probe settings.

Now I have to wait and see if the probe will learn. I'll keep you updated.

Share this post


Link to post
Share on other sites
17 hours ago, Duvak said:

So next is the SNMP scan. My probe is configured to not scan it's own gateway (which is the firewall) because the management IP for the firewall is in a different VLAN. So i've added the gateway IP to the probe settings.

I ran into this with Auvik, but I am sure the same condition will apply.  In order for the gateway to have an ARP record for a device, the device must be communicating through the gateway. So my situation was that Auvik was not finding some thin client MAC addresses in the SAME subnet. Auvik doesn't do it's own ARP scan, it only uses Layer 3 device SNMP info to glean this (when it can't learn it directly with SNMP). Anyways, the thin clients were only communicating on the local subnet, nothing was going to or from the gateway, so the ARP table never had the device information. As soon as I logged into the firewall and performed a local ping to the devices, it cached the ARP info and the next Auvik scan pulled the info right in. Since the probe is pinging to the remote devices through the gateway, as long as they have a route back through the gateway (a default route?) the gateway should learn and cache the ARP info for 10 minutes or whatever, and the probe should be able to learn it. 

I just thought I would mention it.. If the device is quiet and doesn't use the default gateway or respond to IP's outside of the subnet, it can be hard to discover....

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×