Heartbeat monitoring - agent stopped

edited June 2014 in General
Sometimes there's a few servers that give us an error email simular to this:

EVENT # 182278
EVENT LOG Application
EVENT TYPE Error
OPCODE Info
SOURCE EventSentry
CATEGORY Heartbeat Monitoring
EVENT ID 11001
DATE / TIME 6/4/2014 4:49:40 PM
COMPUTERNAME EVENTSENTRYII
MESSAGE Host MAIL (Domain Servers) changed its AGENT status from OK to ERROR. The reason for the status change was: "Agent Stopped".


How can we prevent the agent from stopping? Out of of 26 servers and 3 sensor computers , 4 systems are in "Service stopped" state. I can start the service back up from the ES console, but the next day it will be stopped again.

Comments

  • The first step would be to determine whether the remote agents are being stopped, or whether they are crashing.

    On 2008 you can also open the "View all problem reports" application to see if there are reliability issues.

    You can also review the application and system event logs on the remote machines to see if there is any indication that the EventSentry agent terminated (look for events like "Application Error").

    If there are then we can send you instructions on how to enable crash dumps so that we can investigate the issue.
  • Here is the one example above:

    Event Type: Error
    Event Source: Application Error
    Event Category: (100)
    Event ID: 1000
    Date: 6/5/2014
    Time: 8:16:23 AM
    User: N/A
    Computer: MAIL
    Description:
    Faulting application EVENTSENTRY_SVC.EXE, version 3.0.1.98, faulting module myodbc5w.dll, version 5.2.4.0, fault address 0x0002bc1f.

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
    Data:
    0000: 41 70 70 6c 69 63 61 74 Applicat
    0008: 69 6f 6e 20 46 61 69 6c ion Fail
    0010: 75 72 65 20 20 45 56 45 ure EVE
    0018: 4e 54 53 45 4e 54 52 59 NTSENTRY
    0020: 5f 53 56 43 2e 45 58 45 _SVC.EXE
    0028: 20 33 2e 30 2e 31 2e 39 3.0.1.9
    0030: 38 20 69 6e 20 6d 79 6f 8 in myo
    0038: 64 62 63 35 77 2e 64 6c dbc5w.dl
    0040: 6c 20 35 2e 32 2e 34 2e l 5.2.4.
    0048: 30 20 61 74 20 6f 66 66 0 at off
    0050: 73 65 74 20 30 30 30 32 set 0002
    0058: 62 63 31 66 bc1f

    I am able to restart the service. So for now I just went into the service and in the recovery tab I changed the first, second and subsequent failures from no action to "Restart the Service". Hopefully that keeps it up.
  • Thanks. It looks like it's crashing in the MySQL ODBC driver. You should definitely try to get us a crash dump just in case, although we can't fix any issues in the ODBC driver. I will post instructions.

    MySQL has a version 5.3.2 of their ODBC drivers available - I would try to update one of the machines where the agent keeps crashing to see if that resolves the issues.

    Are you writing to more than one database or just one?
  • Just writing it to the one sql database on the server that is running the main event sentry console.

  • HI Keith,

    Here are some instructions for generating a crash dump:

    http://www.eventsentry.com/kb/257

    We will also need a debug log from the agent that is set to HIGH:

    http://www.eventsentry.com/kb/60

    Once you have both files, you can upload them here:

    https://www.netikus.net/free_uploads.html?SESSION=
  • Thanks I installed the new mysql driver. Needed to also install VC2010+ runtime for that to work. I changed the logging to the higher level. Its still running now but I will check in the morning.
  • Service stayed up on the two problem servers. I wonder if upgrading the mysql driver fixed it.

    Will keep an eye on it!
  • Thanks for the update, we appreciate it. It would be an easy solution if that were the case, let's keep our fingers crossed.

    Please do update us if the agent does not stay running.
  • I have two DIFFERENT servers from this original post that this happened to today. Similar error because it says Faulting Application EVENTSENTRY_SVC.EXE version 3.0.1.98 faulting module: myodbc5w.dll, version 5.2.4.0.

    So my thinking is to update the odbc on these two servers as well.
  • Yes, I'd definitely recommend that. If your computers are in a AD domain then you can also distribute the MySQL ODBC driver using group policy / software distribution.

    Let us know if updating those machines helps as well.
Sign In or Register to comment.