Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”
Hortonworks,
I have 2 installations of HDP 2.6.3 - One installation on a Centos 7 Virtual Machine, and one installation on a CentOS 7 Linux PC.
Both are configured with different static IP addresses and both were set up identically with MySQL 5.7 for Ambari and Hive.
The VM installation always works every single time on boot. Flawlessly. No issues with Stale PIDs, etc. It just works.
HOWEVER, the PC installation of HDP NEVER starts successfully on boot, despite being configured within Ambari to start all services on boot. For example, I always get the same message:
# ambari-server status Using python /usr/bin/python Ambari-server status Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid
After checking the ambari server log at /var/log/ambari-server/ambari-server.log I always see the same error:
BAccessorImpl:119 - Error while creating database accessor
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
There is nothing wrong with the MySQL installation. I can log in and see tables with ambari and hive usernames.
Variations of this same issue "Ambari Server not running, Stale PID" have been posted multiple times throughout this forum. I'm really tired of dealing with this issue with HDP.
PLEASE ADVISE how to fix.
Thank you.
This could be what you are looking for HDFS Support for Multihomed Networks I haven't tried it out myself but its an avenue to adventure!
HTH
Update: It turns out, upon further review, that this entire issue may have to do with networking. Previously, my bare metal installation had connected to my network via WiFi (not a wired connection). When I switched to a Wired connection, obviously, the network connection was established immediately upon boot, and subsequently, ambari-server established a new PID and all was well.
However, I don't want to rely on solely a wired connection for my cluster. How do I order the ambari-server and related to services to start AFTER WiFi networking is established?
PLEASE ADVISE.
Thanks.
Hi @Felix Albani,
Although I strongly believe that sequence/order for services at startup has nothing to do with the "com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure" issue, I am willing to try altering the service order on bare metal IF (and ONLY IF) you can provide clear, step-by-step instructions on how to modify the sequence order within a post on this thread. If you can do this, then I will give it a try. However, please do not include URLs to other sites, as I unfortunately do not have time to research the other links. Again, if you can provide step-by-step instructions on how to modify the sequence order within a post on this thread, I will give it a try.
Please advise. Thanks.
Thank you for your reply, but once again, the boot sequence order has nothing to do with the bare metal dbc4.CommunicationsException: Communications link failure. If it did, the VM would throw the same error since the start sequence is unmodified on that machine, similar to the bare metal instance.
Please see my last reply.
"I actually ran that command during both setups - VM and bare metal. But I did it again at your suggestion and restarted the MySQL service."
In other words, I have run:
# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
during setup and after setup.
No changes. I am still encountering Stale PID.
This issue is super frustrating.... :(
Thanks for the feedback just monitor and update this thread after re-running the mysql-connector -java.jar.
Thanks for your reply. I actually ran that command during both setups - VM and bare metal. But I did it again at your suggestion and restarted the MySQL service.
FYI - I did compare the ambari.properties of both clusters. They are exactly the same. Line for line.
Can you re-run this as the process owner of Ambari server, it could look trivial. Also compare the permissions on they mysql-connector -java.jar on both VM and bare metal !
# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
You are right if you didn't tweak the startup order on the VM then no reason to do that. After running the above please monitor the ambari-server.log what you could do is do a backup of the old logs for analysis. Could you also compare the ambari.properties of both clusters.
Please revert
HTH
Thank you for tyour reply, but that link cannot explain why two different installations configured exactly the same way on the same OS, one works and one doesn't. For example, I didn't have to configure the VM to start services in a different order and it works fine.
This seems to be a major bug in HDP as the question is never really addressed fully in any thread related to this topic.
Perhaps you have already checked this but, have you configured sequence/order for services at startup?
If not please review this link
https://www.putorius.net/2016/04/start-specifc-services-first-in-redhat.html
HTH
Based on the error you shared I think in bare metal server the network link/mysql database maybe taking longer to startup for some reason.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Please review this:
https://access.redhat.com/solutions/18015
Most of the diagnostics for this issue are around network/firewall and database.
As I mentioned in previous post perhaps pushing the ambari service start order to perhaps last place may help. Could you explore the above diagnostics and see if changing the order of startup helps in some way?
