Hortonworks.com
  • Explore
    • All Tags
    • All Questions
    • All Repos
    • All SKB
    • All Articles
    • All Ideas
    • All Users
    • All Badges
    • Leaderboard
  • Create
    • Ask a question
    • Add Repo
    • Create Article
    • Post Idea
  • Tracks
    • All Tracks
    • Community Help
    • Cloud & Operations
    • CyberSecurity
    • Data Ingestion & Streaming
    • Data Processing
    • Data Science & Advanced Analytics
    • Design & Architecture
    • Governance & Lifecycle
    • Hadoop Core
    • Sandbox & Learning
    • Security
    • Solutions
  • Login
HCC Hortonworks Community Connection
  • Home /
  • Sandbox & Learning /
avatar image

Unable to copy from local to HDSF

Question by Matt Jan 05 at 08:37 AM Sandbox

Hi

I am a brand new user. I have installed Hortonworks Sandbox on my VM virtual box manager.

I have logged into Ambari 127.0.0.1 using maria_dev user name.

I have installed Putty and set up by using a connection with maria_dev@sandbox-hdp.

I cannot copy a file from my local directory to HDFS. Do I need to set permissions or have I missed a step in the set up process?

Any assistance would be greatly appreciated.

Thanks

Comment

People who voted for this

0 Show 0
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

13 Replies

· Add your reply
  • Sort: 
  • Votes
  • Created
  • Oldest
avatar image

Answer by Jay Kumar SenSharma · Jan 05 at 08:40 AM

@Matthew May

Can you please share the exact command and the error that you are getting while copying a file from local to HDFS?

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Matt · Jan 05 at 08:44 AM

@ Jay Kumar SenSharma

I have a file on my desktop called sample.txt (in location Users/Matt/dir3/sample.txt.

I have tried this:

hadoop fs -copyFromLocal /Users/Matt/dir3/sample.txt /user/maria_dev/ and receive the error:

copyFromLocal: `/Users/Matt/dir3/sample.txt': No such file or directory

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Jay Kumar SenSharma · Jan 05 at 08:59 AM

Based on the error it looks like you are trying to Push the files from your MAC desktop files to HDP Sandbox HDFS cluster. Please correct me if i am wrong.

copyFromLocal: `/Users/Matt/dir3/sample.txt': No such file or directory

.

Please check few things:

1. Is this file "/Users/Matt/dir3/sample.txt" existing and has correct permission ? so that the user who is running "hadoop" command has proper read access to this file? Please share the output of the following file.

# ls -l /Users/Matt/dir3/sample.txt

2. If you just want to Put the local files to HDFS then another simple approach will be to use Ambari File View.

.

.

Additionally if you want to Put a file inside "/user/maria_dev/" HDFS directory then the user who is running this hadoop command must belong to "hadoop" group (OR) should have username "maria_dev" because the HDFS directory has the following permission.

[root@sandbox /]# su - hdfs -c "hadoop fs -ls /user" | grep maria_dev drwxr-xr-x - maria_dev hadoop 0 2017-10-21 11:01 /user/maria_dev

.

Comment

People who voted for this

0 Show 1 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Jay Kumar SenSharma ♦ · Jan 05 at 09:10 AM 0
Share

Additionally, In addition to the permissions issue. If you want to run the "hadoop" client commands as you posted from your local machine then you will need to make sue that your local machine is setup as a Hadoop Client machine (means it should have all the hadoop libraries + core-site.xml and hdfs-site.xml) files.

.

However as you are using Hortonworks Sandbox hence it will be really good and easy to Put to files to HDFS from your laptop using Ambari File View. Please see: https://hortonworks.com/tutorial/hadoop-tutorial-getting-started-with-hdp/section/2/

avatar image

Answer by Matt · Jan 05 at 09:05 AM

@ Jay Kumar SenSharma

you are correct, I am trying to push the file from my (Windows) desktop to Sandbox. The file does exist, however I am unsure if the correct permissions have been set? The output is:

I understand that I can use Ambari File View, thank you, however my desire is to upload multiple files at once from a directory (the sample file is a test).

ls: cannot access /Users/Matt/dir3/sample.txt: No such file or directory

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Matt · Jan 05 at 09:10 AM

@ Jay Kumar SenSharma


Could you please explain how the user who is running hadoop command can belong to "hadoop" group?

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Jay Kumar SenSharma · Jan 05 at 09:11 AM

If you want to run the "hadoop" client commands as you posted from your Windows machine then you will need to make sue that your local machine is setup as a Hadoop Client machine (means it should have all the hadoop libraries + core-site.xml and hdfs-site.xml) files.

.

However as you are using Hortonworks Sandbox hence it will be really good and easy to Put to files to HDFS from your laptop using Ambari File View. Please see: https://hortonworks.com/tutorial/hadoop-tutorial-getting-started-with-hdp/section/2/

.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Matt · Jan 05 at 09:13 AM

@ Jay Kumar SenSharma

thank you, however I wish to upload multiple files at once from a directory. In Ambari File View, I can only upload a single file if I understand correctly.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Jay Kumar SenSharma · Jan 05 at 09:19 AM

Regarding your query: "Could you please explain how the user who is running hadoop command can belong to "hadoop" group?"

.

For Windows environment i can not help much. However in Linux based environment you can simple use the to add "testuser" to hadoop group in linux based environment.

# sudo adduser --ingroup hadoop testuser

.

If you really want to test the "copyFromLocal" command then you should do it inside the Sandbox instance as it has "hadoop" group already present in it and all the requierd libraries.

Example:

Login to Snadbox using SSH session on port 2222 (this port must be used instead of default SSH port). Or while using Putty please define the SSH port as 2222

# ssh root@127.0.0.1  -p 2222

.

Once you are inside the Sandbox terminal then you try running the command:

# su - maria_dev
# hadoop fs -copyFromLocal /etc/passwd /user/maria_dev/

.

Instead of file "/etc/passwd" you can push your own files to HDFS. You will need to just SCP your files to Sandbox and then from Sandbox you can put them to HDFS.

Or use Ambari File view to directly post your files to HDFS.

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Matt · Jan 05 at 09:27 AM

@ Jay Kumar SenSharma

thank you for your reply.

From Putty I used your command above:

ssh root@127.0.0.1  -p 2222

And received the following error:

ssh: connect to host 127.0.0.1 port 2222: Connection refused

Can you also please explain what you mean by "you will need to SCP your files to Sandbox and then from Sandbox you can put them to HDFS"?

Comment

People who voted for this

0 Show 0 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image

Answer by Jay Kumar SenSharma · Jan 05 at 09:39 AM

@Matt

I am not sure from putty how you entered that command "ssh root@127.0.0.1 -p 2222"

But the best way to connect to Sandbox is described in the following Tutorial which might be really helpful. https://hortonworks.com/tutorial/learning-the-ropes-of-the-hortonworks-sandbox/#terminal-access

.

Usually if you want to put a file from Windows to Linux host then WinScp kind of UI utilities are the best if you want to do SCP.

Comment

People who voted for this

0 Show 2 · Share
10 |6000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Jay Kumar SenSharma ♦ · Jan 06 at 11:16 PM 0
Share

@Matt

Using WInSCP once we are able to copy the file to Sandbox then we should be able to run the following commands to push those files to HDFS from Sandbox.

Suppose in Sandbox host the file is present at "/toot/sample.txt" then it can be pushed to HDFS as following:

.

You will need to do a SSH to the Sandbox first and then run the following commands:

# su - maria_dev
# hadoop fs -copyFromLocal  /root/sample.txt  /user/maria_dev/
(OR)
# hdfs dfs -put  /root/sample.txt  /user/maria_dev/
# hdfs dfs -ls  /user/maria_dev/


NOTE: the user "maria_dev" should have read permission atleast on the file "/root/sample.txt" before it tried to read/push it to HDFS.

.


avatar image Jay Kumar SenSharma ♦ Jay Kumar SenSharma ♦ · Jan 07 at 01:59 AM 0
Share

@Matt

If this issue is resolved then it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.

  • 1
  • 2
  • ›

Your answer

Hint: You can notify a user about this post by typing @username

Up to 5 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

65
Followers
follow question

Answers Answer & comments

HCC Guidelines | HCC FAQs | HCC Privacy Policy

Hortonworks - Develops, Distributes and Supports Open Enterprise Hadoop.

© 2011-2017 Hortonworks Inc. All Rights Reserved.
Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
Privacy Policy | Terms of Service

HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service

© 2011-2018 Hortonworks Inc. All Rights Reserved.

Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

  • Anonymous
  • Login
  • Create
  • Ask a question
  • Add Repo
  • Create SupportKB
  • Create Article
  • Post Idea
  • Tracks
  • Community Help
  • Cloud & Operations
  • CyberSecurity
  • Data Ingestion & Streaming
  • Data Processing
  • Data Science & Advanced Analytics
  • Design & Architecture
  • Governance & Lifecycle
  • Hadoop Core
  • Sandbox & Learning
  • Security
  • Solutions
  • Explore
  • All Tags
  • All Questions
  • All Repos
  • All SKB
  • All Articles
  • All Ideas
  • All Users
  • Leaderboard
  • All Badges