Hortonworks.com
  • Explore
    • All Tags
    • All Questions
    • All Articles
    • All Ideas
    • All Repos
    • All SKB
    • All Users
    • All Badges
    • Leaderboard
  • Create
    • Ask a question
    • Create Article
    • Post Idea
    • Add Repo
  • Tracks
    • All Tracks
    • Community Help
    • Cloud & Operations
    • CyberSecurity
    • Data Ingestion & Streaming
    • Data Processing
    • Data Science & Advanced Analytics
    • Design & Architecture
    • Governance & Lifecycle
    • Hadoop Core
    • Sandbox & Learning
    • Security
    • Solutions
  • Login
HCC Hortonworks Community Connection
  • Home /
  • Data Processing /
avatar image

Always encounter “Ambari Server not running, Stale PID File at: /var/run/ambari-server/ambari-server.pid”

Question by John Doe Jul 05, 2018 at 06:27 PM ambari-servermysql

Hortonworks,

I have 2 installations of HDP 2.6.3 - One installation on a Centos 7 Virtual Machine, and one installation on a CentOS 7 Linux PC.

Both are configured with different static IP addresses and both were set up identically with MySQL 5.7 for Ambari and Hive.

The VM installation always works every single time on boot. Flawlessly. No issues with Stale PIDs, etc. It just works.

HOWEVER, the PC installation of HDP NEVER starts successfully on boot, despite being configured within Ambari to start all services on boot. For example, I always get the same message:

# ambari-server status

Using python  /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid

After checking the ambari server log at /var/log/ambari-server/ambari-server.log I always see the same error:

BAccessorImpl:119 - Error while creating database accessor
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

There is nothing wrong with the MySQL installation. I can log in and see tables with ambari and hive usernames.

Variations of this same issue "Ambari Server not running, Stale PID" have been posted multiple times throughout this forum. I'm really tired of dealing with this issue with HDP.

PLEASE ADVISE how to fix.

Thank you.

Comment

People who voted for this

0 Show 0
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

10 Replies

· Add your reply
  • Sort: 
  • Votes
  • Created
  • Oldest
avatar image

Answer by Geoffrey Shelton Okot · Jul 09, 2018 at 09:46 PM

@John Doe

This could be what you are looking for HDFS Support for Multihomed Networks I haven't tried it out myself but its an avenue to adventure!

HTH

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by John Doe · Jul 09, 2018 at 08:53 PM

@Felix Albani, @Geoffrey Shelton Okot

Update: It turns out, upon further review, that this entire issue may have to do with networking. Previously, my bare metal installation had connected to my network via WiFi (not a wired connection). When I switched to a Wired connection, obviously, the network connection was established immediately upon boot, and subsequently, ambari-server established a new PID and all was well.

However, I don't want to rely on solely a wired connection for my cluster. How do I order the ambari-server and related to services to start AFTER WiFi networking is established?

PLEASE ADVISE.

Thanks.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by John Doe · Jul 09, 2018 at 12:07 AM

Hi @Felix Albani,

Although I strongly believe that sequence/order for services at startup has nothing to do with the "com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure" issue, I am willing to try altering the service order on bare metal IF (and ONLY IF) you can provide clear, step-by-step instructions on how to modify the sequence order within a post on this thread. If you can do this, then I will give it a try. However, please do not include URLs to other sites, as I unfortunately do not have time to research the other links. Again, if you can provide step-by-step instructions on how to modify the sequence order within a post on this thread, I will give it a try.

Please advise. Thanks.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by John Doe · Jul 08, 2018 at 02:48 AM

@Felix AlbaniT

Thank you for your reply, but once again, the boot sequence order has nothing to do with the bare metal dbc4.CommunicationsException: Communications link failure. If it did, the VM would throw the same error since the start sequence is unmodified on that machine, similar to the bare metal instance.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by John Doe · Jul 06, 2018 at 01:20 PM

Hi @Geoffrey Shelton Okot

Please see my last reply.

"I actually ran that command during both setups - VM and bare metal. But I did it again at your suggestion and restarted the MySQL service."

In other words, I have run:

  # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

during setup and after setup.

No changes. I am still encountering Stale PID.

This issue is super frustrating.... :(

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by Geoffrey Shelton Okot · Jul 06, 2018 at 07:23 AM

@John Doe

Thanks for the feedback just monitor and update this thread after re-running the mysql-connector -java.jar.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by John Doe · Jul 05, 2018 at 10:00 PM

Hi @Geoffrey Shelton Okot

Thanks for your reply. I actually ran that command during both setups - VM and bare metal. But I did it again at your suggestion and restarted the MySQL service.

FYI - I did compare the ambari.properties of both clusters. They are exactly the same. Line for line.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by Geoffrey Shelton Okot · Jul 05, 2018 at 07:38 PM

@John Doe

Can you re-run this as the process owner of Ambari server, it could look trivial. Also compare the permissions on they mysql-connector -java.jar on both VM and bare metal !

#  ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

You are right if you didn't tweak the startup order on the VM then no reason to do that. After running the above please monitor the ambari-server.log what you could do is do a backup of the old logs for analysis. Could you also compare the ambari.properties of both clusters.

Please revert

HTH

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by John Doe · Jul 05, 2018 at 07:00 PM

@Felix Albani

Thank you for tyour reply, but that link cannot explain why two different installations configured exactly the same way on the same OS, one works and one doesn't. For example, I didn't have to configure the VM to start services in a different order and it works fine.

This seems to be a major bug in HDP as the question is never really addressed fully in any thread related to this topic.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image

Answer by Felix Albani · Jul 05, 2018 at 06:46 PM

@John Doe

Perhaps you have already checked this but, have you configured sequence/order for services at startup?

If not please review this link

https://www.putorius.net/2016/04/start-specifc-services-first-in-redhat.html

HTH

Comment

People who voted for this

0 Show 1 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

avatar image Felix Albani · Jul 06, 2018 at 01:47 PM 0
Share
@John Doe

Based on the error you shared I think in bare metal server the network link/mysql database maybe taking longer to startup for some reason.

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Please review this:

https://access.redhat.com/solutions/18015

Most of the diagnostics for this issue are around network/firewall and database.

As I mentioned in previous post perhaps pushing the ambari service start order to perhaps last place may help. Could you explore the above diagnostics and see if changing the order of startup helps in some way?

Your answer

Hint: You can notify a user about this post by typing @username

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

70
Followers

Answers Answer & comments

This website uses cookies for analytics, personalisation and advertising. To learn more or change your cookie settings, please read our Cookie Policy. By continuing to browse, you agree to our use of cookies.

HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service

© 2011-2019 Hortonworks Inc. All Rights Reserved.

Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Login
  • Create
  • Ask a question
  • Create Article
  • Post Idea
  • Add Repo
  • Create SupportKB
  • Tracks
  • Community Help
  • Cloud & Operations
  • CyberSecurity
  • Data Ingestion & Streaming
  • Data Processing
  • Data Science & Advanced Analytics
  • Design & Architecture
  • Governance & Lifecycle
  • Hadoop Core
  • Sandbox & Learning
  • Security
  • Solutions
  • Explore
  • All Tags
  • All Questions
  • All Articles
  • All Ideas
  • All Repos
  • All SKB
  • All Users
  • Leaderboard
  • All Badges