This notebook was prepared by Donne Martin. Source and license info is on GitHub.

HDFS

Run an HDFS command:

!hdfs

Run a file system command on the file systems (FsShell):

!hdfs dfs

List the user's home directory:

!hdfs dfs -ls

List the HDFS root directory:

!hdfs dfs -ls /

Copy a local file to the user's directory on HDFS:

!hdfs dfs -put file.txt file.txt

Display the contents of the specified HDFS file:

!hdfs dfs -cat file.txt

Print the last 10 lines of the file to the terminal:

!hdfs dfs -cat file.txt | tail -n 10

View a directory and all of its files:

!hdfs dfs -cat dir/* | less

Copy an HDFS file to local:

!hdfs dfs -get file.txt file.txt

Create a directory on HDFS:

!hdfs dfs -mkdir dir

Recursively delete the specified directory and all of its contents:

!hdfs dfs -rm -r dir

Specify HDFS file in Spark (paths are relative to the user's home HDFS directory):

data = sc.textFile ("hdfs://hdfs-host:port/path/file.txt")