Read/Write single file in DataBricks
I have a file which contains a list of names stored in a simple text file. Each row contains one name. Now I need to pro grammatically append a new name to this file based on a users input. For the input itself I use DataBricks widgets - this is working just fine and I have the new name stored in a string object. Now I need to append this name to my file.
the file is mounted in the DataBricks File System (DBFS) under /mnt/blob/myNames.txt
when trying to read the file like this:
f = open("/mnt/blob/myNames.txt", "r")
print f
it returns an error "No such file or directory"
So I tried to wrap my new name into a dataframe and append it to the existing file but this also did not work as dataframe.write.save is designed to write into folders
what would be the most simple python could that I could use to append this new name to my file?
Answer
You can write and read files from DBFS with dbutils. Use the dbutils.fs.help() command in databricks to access the help menu for DBFS.
You would therefore append your name to your file with the following command:
dbutils.fs.put("/mnt/blob/myNames.txt", new_name)
You are getting the "No such file or directory" error because the DBFS path is not being found. Use dbfs:/ to access a DBFS path. This is how you should have read the file:
f = open("/dbfs/mnt/blob/myNames.txt", "r")