Difference between revisions of "OPS435 Python Lab 5"

From CDOT Wiki
Jump to: navigation, search
(INVESTIGATION 1: Files)
 
(164 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 +
<font color='red'>
 +
'''** DO NOT USE - TO BE UPDATED FOR CENTOS 8.0 **'''
 +
</font>
 
= LAB OBJECTIVES =
 
= LAB OBJECTIVES =
:This lab will introduce object oriented programming, using classes to create objects, and reading/writing to files.
 
  
:The first investigation will focus on file management, opening files, saving data to files, and reading files. Since many tasks that system administrators deal with files, this is a crucial skill to understand. The second investigation will cover object oriented programming. Objects are primarily used to store data and code.
+
:So far, you have created Python scripts to prompt a user to input data from the keyboard. When creating Python scripts, you may also need to be able to process large volumes of information, or store processed data for further processing. The first investigation in this lab will focus on file management, opening files, saving data to files, and reading files.
 +
 
 +
:'''NOTE:''' Since many tasks that system administrators perform deal with files, this is a crucial skill to understand.
 +
 
 +
:It is very important to provide logic in your Python script in case it '''encounters an error'''. An example would be an invalid path-name or trying to close a file that is already closed. The second investigation in this lab will look into how the Python interpreter '''handle errors''' (commonly referred to as "exception handling") at run time, and learn how to write Python code that will run gracefully even when problems occur during program execution.
 +
 
  
 
== PYTHON REFERENCE ==
 
== PYTHON REFERENCE ==
:As you develop your Python scripting skills, you may start to be "overwhelmed" with the volume of information that you have absorbed over these labs. One way to help, is to write what you have learned in your labs into your notes. Also, in programming, it is important to use online references in order to obtain information regarding Python scripting techniques and tools.
+
:In previous labs, you have been advised to make notes and use online references. This also relates to working with files and learning about objected oriented programming. You may be "overwhelmed" with the volume of information involved in this lab.  
  
 
:Below is a table with links to useful online Python reference sites (by category). You may find these references useful when performing assignments, etc.
 
:Below is a table with links to useful online Python reference sites (by category). You may find these references useful when performing assignments, etc.
  
https://docs.python.org/3/tutorial/errors.html
+
{| class="wikitable" | style="margin-left:20px; border: 2px solid black;"
https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files
+
|- style="border: 2px solid black;font-weight:bold;text-align:center;"
https://docs.python.org/3/library/functions.html#open
+
| style="border: 2px solid black;" | Category
https://docs.python.org/3/tutorial/classes.html
+
| style="border: 2px solid black;" | Resource Link
https://docs.python.org/3/library/exceptions.html
+
 
 +
|- style="background-color:white;border:none;"
 +
| style="border: 2px solid black;" valign="top"| 
 +
:Handling Errors &amp; Exceptions
 +
| style="border: 2px solid black;" valign="top"| 
 +
:[https://docs.python.org/3/tutorial/errors.html Errors &amp; Exceptions]
  
= INVESTIGATION 1: Files =
+
|-  style="background-color:white;border:none;"
:This first investigation will cover the ability to open text files on the system, read the contents, store the contents, and finally write the contents back into a file. This is a very common practice in programming, used in situations such as logging output, logging errors, reading/creating configuration files, temporary files, and more. Files are accessed through the use of file objects, an object is a storage location which stores data in the form of attributes(variables) and methods(functions). Creating our own objects will be covered in investigation 3.  
+
| style="border: 2px solid black;" valign="top"| 
 +
:Built-in Exceptions
 +
| style="border: 2px solid black;" valign="top"| 
 +
:[https://docs.python.org/3/library/exceptions.html Built-in Exceptions]
  
== PART 1 - Reading From Files ==
+
|-  style="background-color:white;border:none;"
 +
| style="border: 2px solid black;" valign="top"| 
 +
:Data Manipulation via Files
 +
| style="border: 2px solid black;" valign="top"| 
 +
:[https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files Reading &amp; Writing Files]
 +
 
 +
|-  style="background-color:white;border:none;"
 +
| style="border: 2px solid black;" valign="top"| 
 +
:Opening Files
 +
| style="border: 2px solid black;" valign="top"| 
 +
:[https://docs.python.org/3/library/functions.html#open Open File Usage]
 +
 
 +
|-  style="background-color:white;border:none;"
 +
| style="border: 2px solid black;" valign="top"| 
 +
:Using Classes
 +
| style="border: 2px solid black;" valign="top"| 
 +
:[https://docs.python.org/3/tutorial/classes.html Classes]
 +
|}
 +
 
 +
= INVESTIGATION 1: Working with Files =
 +
:You will now learn how to write Python scripts in order to open text files, to read the contents within a text file, to process the contents, and finally to write the processed contents back into a file. These operations are very common, and are used extensively in programming. Examples of file operations would include situations such as logging output, logging errors, reading and creating configuration/temporary files, etc.
 +
 
 +
:Files are accessed through the use of '''file objects'''. An '''object''' is a '''storage location''' which stores data in the form of '''attributes''' (variables) and '''methods''' (functions). Creating our own objects will be covered later in investigation 3.
 +
 
 +
== PART 1 - Reading Data From Files ==
 
:'''Perform the Following Steps:'''
 
:'''Perform the Following Steps:'''
::#To start, open the ipython3 shell:<source lang="python">
+
:#Create a new python file for testing.
ipython3
+
:#Create a new text file in the '''lab5''' directory:<source lang="bash">
</source>
+
cd ~/ops435/lab5
::#Create a new text file in the lab5 directory:<source lang="python">
+
vim ~/ops435/lab5/data.txt
%cd ~/ops435/lab5
 
%vim ~/ops435/lab5/data.txt
 
 
</source>  
 
</source>  
::#Place the following content inside the new file and save it:<source lang="python">
+
:#Place the following content inside the new text file and save it:<source>
 
Hello World
 
Hello World
 
This is the second line
 
This is the second line
 
Third line
 
Third line
 
Last line
 
Last line
</source>
+
</source><br>In order to read data from a text file, we need to create an object that will be used to access the data in a file. In some programming languages (like C) this is called a file descriptor, or a file pointer. In Python, it's an '''object'''.<br><br>
::#Now lets write some python code to open this file for reading.<source lang="python">
+
:#Now lets write some python code to open this created file for reading. We will define and object called '''"f"''' in order to help retrieve content from our text file. Issue the following:<source lang="python">
 
f = open('data.txt', 'r')
 
f = open('data.txt', 'r')
 +
</source><br>The '''open()''' function takes two string arguments: a path to a file, and a mode option (to ask for reading, writing, appending, etc). The ''open()'' function will return a file object to us, this file object will allow us to read the lines inside the file.<br><br>
 +
:#Here are the most useful functions for text file manipulation:<source lang="python">
 +
f.read()            # read all lines and stores in a string
 +
f.readlines()      # read all lines and stores in a list
 +
f.readline()        # read first line, if run a second time it will read the second line, then third
 +
f.close()          # close the opened file
 
</source>
 
</source>
::#The f is a new variable we are creating to store the file object. The open() function takes two string arguments, a path to a file, and a mode option for reading, writing, appending, etc. The open() function will return a special object to us, this object will allow us to read the lines inside the file. Before we use this object lets inspect it with dir().<source lang="python">
+
:#Next, read data from the buffer of the opened file and store the contents into a variable called <code>read_data</code>, and then confirm the contents of the variable <code>read_data</code>:<source lang="python">
dir(f)
 
</source>
 
::#Lets inspect some of the functions we can use on this file object.<source lang="python">
 
help(f.read)            # Reads all lines and stores in a string
 
help(f.readlines)      # Reads all lines and stores in a list
 
help(f.readline)        # Reads first line, if run a second time it will read the second line, then third
 
help(f.writable)     
 
help(f.close)         
 
f.writable()            # Object method (function returns value)
 
f.name                  # Object attribute (variable string)
 
f.closed                # Object attribute (variable boolean)
 
</source>
 
::#Next read data from the file and close the file to free up resources.<source lang="python">
 
 
read_data = f.read()
 
read_data = f.read()
read_data
+
print(read_data)
</source>
+
</source><br>After you have completed accessing data within a file, you should '''close''' the file in order to free up the computer resources. It is sometimes useful to first confirm that the file is still open prior to closing it. But really you should know - it's your code that would have opened it:<source lang="python">
::#Finally close the file to free up the computer resources. First lets check to see if the file is already closed.<source lang="python">
 
f.closed                                # This object attribute will give true if the file is closed and false if open
 
 
f.close()                              # This method will close the file
 
f.close()                              # This method will close the file
f.closed                                # Confirm the file is closed
+
</source><br>Let's take a moment to revisit the '''file read''' sequence. The following code sequence will open a file, store the contents of a file into a variable, close the file and provide confirmation that the file has been closed:<source lang="python">
</source>
 
::#All together the code for reading from a file should look like the following:<source lang="python">
 
 
f = open('data.txt', 'r')              # Open file
 
f = open('data.txt', 'r')              # Open file
 
read_data = f.read()                    # Read from file
 
read_data = f.read()                    # Read from file
f.close()                              # Close file
+
f.close()                              # Close file</source>
f.closed                                # Confirm file is closed
+
:#read_data in this case contains the data from the file in a single <u>long</u> string. The end of each line in the file will show the special character ''''\n'''' which represents the '''newline character''' in a file used to separate lines (or records in a traditional "flat database file"). It would be convenient to '''split''' the line on the new-line characters, so each line can be stored as an item in a list.<br><br>
</source>
+
:#Store the contents of our file into a list called '''list_of_lines''':<source lang="python">
::#Another way to read data from a file is using the with statement. The advantage here is that it will automatically close() the file when complete:<source lang="python">
 
with open('data.txt', 'r') as f:        # Open file
 
    read_data = f.read()                # Read from file
 
f.closed                                # Confirm file is closed
 
</source>
 
::#Next lets read the data<source lang="python">
 
read_data
 
</source>
 
::#This displays the data from the file in a single long string. The end of each line in the file will show a '\n' which represents the newline character in a file. Split the line on the new-line characters, so more inspection can be done on a single line at a time.<source lang="python">
 
dir(read_data)
 
help(read_data.split)
 
 
read_data.split('\n')                      # Returns a list
 
read_data.split('\n')                      # Returns a list
 
list_of_lines = read_data.split('\n')      # Saves returned list in variable
 
list_of_lines = read_data.split('\n')      # Saves returned list in variable
list_of_lines
+
print(list_of_lines)
</source>
+
</source><br>Although the above sequence works, there are '''functions''' and '''methods''' the we can use with '''our object (called "f")''' to place lines from our file into a '''list'''. This would help to reduce code and is considered a more common method to store multiple lines or records within a list.<br><br>
::#The above works, but it's not the best way to get a list of lines. The man benefit above is that the lines can be split in any way desired(on spaces, on periods, etc). The easiest way to just get a list of all lines can be done using the file object and it's methods.<source lang="python">
+
:#Try these two different means to store data into a list more efficiently:<source lang="python">
 +
# METHOD 1:
 +
 
 
f = open('data.txt', 'r')
 
f = open('data.txt', 'r')
 
method1 = list(f)
 
method1 = list(f)
 
f.close()
 
f.close()
method1
+
print(method1)
 +
 
 +
# METHOD 2:
 +
 
 
f = open('data.txt', 'r')
 
f = open('data.txt', 'r')
 
method2 = f.readlines()
 
method2 = f.readlines()
 
f.close()
 
f.close()
method2
+
print(method2)
</source>
 
::#Finally, sometimes all that needs to be done is to print the file to the screen. In these simple cases, there is no need to create extra/large variables to store the data, instead just print it to the screen one line at a time. Doing one of the following methods will save computer resources while reading files.<source lang="python">
 
f = open('data.txt', 'r')
 
for line in f:
 
    print(line, end='')
 
f.close()
 
</source>
 
::#The python print() function by default adds the new-line character to the end of the line. Using the end='' argument used in print replaces the '\n' at the end with nothing ''. This allows the print() to use the new-line characters found on each line of the file that was read. Though if desired you can always strip the new-line characters from any lines. The strip() function will remove all leading and trailing whitespace, which may help in processing some lines or data.<source lang="python">
 
f = open('data.txt', 'r')
 
for line in f:
 
    print(line.strip())
 
f.close()
 
 
</source>
 
</source>
  
 
=== Create a Python Script Demonstrating Reading Files ===
 
=== Create a Python Script Demonstrating Reading Files ===
:'''Perform the Following Instructions'''
+
:#Create the '''~/ops435/lab5/lab5a.py''' script.   
::#Create the '''~/ops435/lab5/lab5a.py''' script.   
+
:#Use the following as a template:<source lang="python">
::#Use the following as a template:<source lang="python">
 
 
#!/usr/bin/env python3
 
#!/usr/bin/env python3
  
Line 120: Line 127:
 
</source>
 
</source>
  
:::*The read_file_string() function should return a string
+
::*This Python script will read the same file ('''data.txt''') that you previously created
:::*The read_file_list() function should return a list
+
::*The '''read_file_string()''' function should return a string
:::*The read_file_list() function must remove the new-line characters from each line in the list
+
::*The '''read_file_list()''' function should return a list
:::*Both functions must accept one argument which is a string
+
::*The '''read_file_list()''' function must remove the new-line characters from each line in the list
:::*The script should show the exact output as the samples
+
::*'''Both functions must accept one argument''' which is a string
:::*The script should contain no errors
+
::*The script should show the exact output as the samples
 +
::*The script should contain no errors
  
::::'''Sample Run 1:'''<source lang="python">
+
:::'''Sample Run 1:'''<source>
 
python3 lab5a.py  
 
python3 lab5a.py  
 
Hello World
 
Hello World
Line 136: Line 144:
 
</source>
 
</source>
  
::::'''Sample Run 2 (with import):'''<source lang="python">
+
:::'''Sample Run 2 (with import):'''<source>
 
import lab5a         
 
import lab5a         
 
file_name = 'data.txt'                             
 
file_name = 'data.txt'                             
lab5a.read_file_string(file_name)
+
print(lab5a.read_file_string(file_name))
'Hello World\nThis is the second line\nThird line\nLast line\n'
+
# Will print 'Hello World\nThis is the second line\nThird line\nLast line\n'
lab5a.read_file_list(file_name)
+
print(lab5a.read_file_list(file_name))
['Hello World', 'This is the second line', 'Third line', 'Last line']
+
# Will print ['Hello World', 'This is the second line', 'Third line', 'Last line']
 
</source>
 
</source>
:::3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.<source lang="bash">
+
::3. Download the checking script and check your work. Enter the following commands from the bash shell.<source lang="bash">
 
cd ~/ops435/lab5/
 
cd ~/ops435/lab5/
 
pwd #confirm that you are in the right directory
 
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab5.py
+
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
 
python3 ./CheckLab5.py -f -v lab5a
 
python3 ./CheckLab5.py -f -v lab5a
 
</source>
 
</source>
:::4. Before proceeding, make certain that you identify any and all errors in lab5a.py. When the checking script tells you everything is OK before proceeding to the next step.
+
::4. Before proceeding, make certain that you identify all errors in lab5a.py. When the checking script tells you everything is OK - proceed to the next step.
 
<br><br>
 
<br><br>
 +
 
== PART 2 - Writing To Files ==
 
== PART 2 - Writing To Files ==
There is a danger involved in this next section, when opening files for writing, Python assumes content in the file is no longer wanted and it's immediately deleted. If the file contents are important, append to the file instead.
+
 
:'''Perform the Following Steps:'''
+
:Up to this point, you have learned how to access text from a file. In this section, you will learn how to write text to a file. Writing data to a file is useful for creating new content in a file or updating (modifying) existing data contained within a file.
::#To start, open the ipython3 shell:<source lang="python">
+
 
ipython3
+
<blockquote style="margin-left:35px;">{{Admon/caution|style="padding-left:25px"|Risk of Losing File Contents|A common problem that new Python programmers may encounter is to accidentally erase existing contents of a file when writing new data to a file. When opening files for writing (using the ''''w'''' open function option), Python assumes existing content in the file is no longer wanted and it's immediately deleted; therefore, if you wish to write data to a file but keep existing content, you need to use the open file option ''''a'''' (append new data to a file).}}</blockquote>
</source>
+
:#When opening a file for writing, the ''''w'''' option is specified with the '''open()''' function. When the 'w' option is specified - previous (existing) content inside the file is deleted. This deletion takes place the moment the open() function is executed, not when writing to the file. If the file that is being written to doesn't exist, the file will be created upon the file opening process.<br><br>
::#When opening a file the 'w' option can be give to open(). When 'w' is provided the previous contents inside the file are deleted, this deleting takes place the moment the open() function is executed, not during the write. If the file that is being written to doesn't exit, it will be created.<source lang="python">
+
:#Create a temporary Python file and open a non-existent data file (called file1.txt) for writing:<source lang="python">
 
f = open('file1.txt', 'w')
 
f = open('file1.txt', 'w')
 
</source>
 
</source>
::#To add lines of text to the file, use the write() method for the file object. For safe file management, always end every line with a '\n'. Multiple lines may also be placed inside a single write, simply put the '\n' wherever a line should end.<source lang="python">
+
:#To confirm that the new file now exists and is empty, issue the following shell command:<source lang="bash">
 +
ls -l file1.txt
 +
</source>To add lines of text to the file, you can use the '''write()''' method for the '''file object'''. Typically you end every line in a text file with the special character ''''\n'''' to represent a "new line". Multiple lines may also be placed inside a single write operation: simply put the special character ''''\n'''' wherever a line should end.<br><br>
 +
:#Try adding multiple lines:<source lang="python">
 
f.write('Line 1\nLine 2 is a little longer\nLine 3 is too\n')
 
f.write('Line 1\nLine 2 is a little longer\nLine 3 is too\n')
</source>
+
</source>Once the '''write()''' method has completed, the final step is to '''close()''' the file. The file MUST be closed properly or else data will not consistently be written to the file. '''NOTE: Not closing a file can lead to corrupted or missing file contents.''':<br><source lang="python">
::#Once the write() has been run, the final step would be to close() the file. The file MUST be closed properly or else data will not consistently be written to the file. Not closing a file can lead to corruption or not changes being made. <source lang="python">
 
 
f.close()
 
f.close()
 
</source>
 
</source>
::#View the contents of the file to make sure the write data was saved. We can use the ipython bash alias %cat to do this.<source lang="python">
+
:#View the contents of the file in the shell to make sure the data was written successfully:<source lang="bash">
%cat file1.txt
+
cat file1.txt
Line 1
+
</source>You will now create a new file called file2.txt, but this time run multiple write() methods in sequence. You will often write to a file multiple times inside a loop:<br><source lang="python">
Line 2 is a little longer
 
Line 3 is too
 
</source>
 
::#Create another file but this time, run multiple write() methods together. The ability to write() multiple lines like this allows for writes to take place inside loops and more complex programs to continuously write to a file.<source lang="python">
 
 
f = open('file2.txt', 'w')
 
f = open('file2.txt', 'w')
f.write('Line 1\nLine 2 is a little longer\nLine 3 is too\n')
+
f.write('Line 1\nLine 2 is a little longer\nLine 3 is as well\n')
 
f.write('This is the 4th line\n')
 
f.write('This is the 4th line\n')
 
f.write('Last line in file\n')
 
f.write('Last line in file\n')
 
f.close()
 
f.close()
 
</source>
 
</source>
::#View the contents of the file2.txt to make sure the write data was saved.<source lang="python">
+
:#Issue the following shell command to confirm that the contents were written to file2.txt:<source lang="bash">
%cat file2.txt
+
cat file2.txt
Line 1
 
Line 2 is a little longer
 
Line 3 is too
 
This is the 4th line
 
Last line in file         
 
 
</source>
 
</source>
::#View the contents of the file1.txt to make sure it has the original data and has not been changed<source lang="python">
+
<blockquote style="margin-left:35px;">{{Admon/important|style="padding-left:25px"|Make Backup Copies of Your Data Files|Since you might make a mistake, and accidentally destroy file contents when writing to your file, it is highly recommended to make backup copies of your files prior to running your Python scripts. This can be particularly useful when performing any future assignment involving writing to files.}}</blockquote>
%cat file1.txt
+
<blockquote style="margin-left:35px;">{{Admon/caution|style="padding-left:25px"|Make Backup Copies of ALL your files|Since you are now writing code that opens files for writing you may accidentally truncate the wrong file (like your assignment file, for example). Make regular backups of all your work. Just copy it to a USB stick or if you're bored - learn version control.}}</blockquote><br>
Line 1
+
<ol style="margin-left:80px;"><li value="7">Issue the following shell commands to backup both of your newly-created files and confirm backup:<source lang="bash">
Line 2 is a little longer
+
cp file1.txt file1.txt.bk
Line 3 is too         
+
cp file2.txt file2.txt.bk
 +
ls -l file*         
 +
</source></li>
 +
<li>Let's demonstrate what can happen if you perform an incorrect write() operation:<source lang="python">
 +
f = open('file2.txt', 'w')
 
</source>
 
</source>
::#In the event that the data in the file is important and should not be overwritten, we can append data to the end of the file instead. Use the option 'a' instead of 'w' to perform appending.<source lang="python">
+
<source lang="bash">
 +
cat file2.txt
 +
</source>You should notice that the previous content in your file2.txt file was destroyed. Why do you you think the previous data is no longer there?<br></li>
 +
<li>Restore your file from the backup and verify the backup restoration:<source lang="bash">
 +
cp file2.txt.bk file2.txt
 +
cat file2.txt
 +
</source>To avoid overwriting the contents of a file, we can '''append''' data to the end of the file instead. Use the option 'a' instead of 'w' to perform appending:<br><br><source lang="python">
 
f = open('file1.txt', 'a')
 
f = open('file1.txt', 'a')
 
f.write('This is the 4th line\n')
 
f.write('This is the 4th line\n')
 
f.write('Last line in file\n')
 
f.write('Last line in file\n')
 
f.close()
 
f.close()
</source>  
+
</source>
::#The final point to make when writing to files is to make sure that the values being written are strings. This means that before trying to place integers, floats, lists, or dictionaries into a file, first either convert the value using str() or extract the specific strings from items in the list. <source lang="python">
+
<source lang="bash">
 +
cat file1.txt
 +
</source>The final thing to consider when writing to files is to make certain that the values being written are '''strings'''. This means that before trying to place integers, floats, lists, or dictionaries into a file, first either convert the value using '''str()''' function or extract the specific strings from items in the list.<br><br></li>
 +
<li>In this example we convert a single number and all the numbers in a list to strings before writing them to a file:<source lang="python">
 
my_number = 1000
 
my_number = 1000
 
my_list = [1,2,3,4,5]
 
my_list = [1,2,3,4,5]
Line 208: Line 223:
 
     f.write(str(num) + '\n')
 
     f.write(str(num) + '\n')
 
f.close()
 
f.close()
</source>
+
</source></li>
::#View the contents of the file3.txt to make sure the write data was saved.<source lang="python">
+
<li>Confirm that the '''write()''' operation was successful<source lang="bash">
%cat file3.txt      
+
cat file3.txt
1000                         
+
</source></li>
1                           
+
</ol>
2                           
+
 
3                           
 
4                           
 
 
</source>
 
 
=== Create a Python Script Demonstrating Writing to Files ===
 
=== Create a Python Script Demonstrating Writing to Files ===
:'''Perform the Following Instructions'''
+
:#Copy '''~/ops435/lab5/lab5a.py''' script to '''~/ops435/lab5/lab5b.py''' script (We need the previous read functions that you created).
::#Copy '''~/ops435/lab5/lab5a.py''' script to '''~/ops435/lab5/lab5b.py''' script. We need the previous read functions written.
+
:#Add the following functions below the two functions that you already created:<source lang="python">
::#Use the following as a template:<source lang="python">
 
#!/usr/bin/env python3
 
 
 
def read_file_string(file_name):
 
    # This is the same code from lab5a.py
 
 
 
def read_file_list(file_name):
 
    # This is the same code from lab5a.py
 
 
 
 
def append_file_string(file_name, string_of_lines):
 
def append_file_string(file_name, string_of_lines):
 
     # Takes two strings, appends the string to the end of the file
 
     # Takes two strings, appends the string to the end of the file
Line 238: Line 240:
 
def copy_file_add_line_numbers(file_name_read, file_name_write):
 
def copy_file_add_line_numbers(file_name_read, file_name_write):
 
     # Takes two strings, reads data from first file, writes data to new file, adds line number to new file
 
     # Takes two strings, reads data from first file, writes data to new file, adds line number to new file
 
+
</source>
 +
:# '''Replace''' the main section of your Python script near the bottom with the following:<source lang="python">
 
if __name__ == '__main__':
 
if __name__ == '__main__':
 
     file1 = 'seneca1.txt'
 
     file1 = 'seneca1.txt'
Line 251: Line 254:
 
     copy_file_add_line_numbers(file2, file3)
 
     copy_file_add_line_numbers(file2, file3)
 
     print(read_file_string(file3))
 
     print(read_file_string(file3))
      
+
     </source>
 +
 
 +
::'''append_file_string():'''
 +
:::#Takes '''two string arguments'''
 +
:::#'''Appends''' to the file(Argument 1) all data from the string(Argument 2)
 +
::'''write_file_list():'''
 +
:::#Takes two arguments: a '''string''' and a '''list'''
 +
:::#'''Writes''' to file(Argument 1) all lines of data found in the list(Argument 2)
 +
::'''copy_file_add_line_numbers():'''
 +
:::#Takes two arguments: '''Both are files path-names''' (which happen to be strings)
 +
:::#'''Reads''' all data from first file(Argument 1), and writes all lines into second file(Argument 2) '''adding''' line numbers
 +
:::#Line numbers should be added to the '''beginning''' of each line with a colon next to them(see sample output below for reference)
 +
:::#'''Hint:''' Use an extra variable for the line number
 +
 
 +
:::'''Sample Run 1:'''<source>
 +
rm seneca1.txt seneca2.txt seneca3.txt
 +
./lab5b.py
 +
First Line
 +
Second Line
 +
Third Line
 +
 
 +
Line 1
 +
Line 2
 +
Line 3
 +
 
 +
1:Line 1
 +
2:Line 2
 +
3:Line 3
  
 
</source>
 
</source>
  
:::'''append_file_string()'''
+
:::'''Sample Run 2 (run second time):'''<source>
::::#Takes two string arguments
 
::::#Appends to the file(Argument 1) all data from the string(Argument 2)
 
:::'''write_file_list()'''
 
::::#Takes two arguments, a string and a list
 
::::#Writes to file(Argument 1) all lines of data found in the list(Argument 2)
 
:::'''copy_file_add_line_numbers()'''
 
::::#Takes two arguments, both strings, both filesnames
 
::::#Reads all data from first file(Argument 1), writes all lines into secon file(Argument 2) adding line numbers
 
::::#Line numbers should be added to the beginning of each line with a colon next to them(see sample output)
 
::::#Hint: review len() and range() functions from lab 3 and lab 4
 
 
 
::::'''Sample Run 1:'''<source lang="python">
 
 
python3 lab5b.py  
 
python3 lab5b.py  
 +
First Line
 +
Second Line
 +
Third Line
 
First Line
 
First Line
 
Second Line
 
Second Line
Line 283: Line 304:
 
</source>
 
</source>
  
::::'''Sample Run 2 (with import):'''<source lang="python">
+
:::'''Sample Run 3 (with import):'''<source>
 
import lab5b
 
import lab5b
 
file1 = 'seneca1.txt'
 
file1 = 'seneca1.txt'
Line 294: Line 315:
  
 
lab5b.read_file_string(file1)
 
lab5b.read_file_string(file1)
'First Line\nSecond Line\nThird Line\nFirst Line\nSecond Line\nThird Line\n'
+
# Will print 'First Line\nSecond Line\nThird Line\nFirst Line\nSecond Line\nThird Line\n'
  
 
lab5b.write_file_list(file2, list1)
 
lab5b.write_file_list(file2, list1)
  
 
lab5b.read_file_string(file2)
 
lab5b.read_file_string(file2)
'Line 1\nLine 2\nLine 3\n'
+
# Will print 'Line 1\nLine 2\nLine 3\n'
  
 
lab5b.copy_file_add_line_numbers(file2, file3)
 
lab5b.copy_file_add_line_numbers(file2, file3)
  
 
lab5b.read_file_string(file3)
 
lab5b.read_file_string(file3)
'1:Line 1\n2:Line 2\n3:Line 3\n'
+
# Will print '1:Line 1\n2:Line 2\n3:Line 3\n'
 
</source>
 
</source>
  
:::3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.<source lang="bash">
+
::3. Download the checking script and check your work. Enter the following commands from the bash shell.<source lang="bash">
 
cd ~/ops435/lab5/
 
cd ~/ops435/lab5/
 
pwd #confirm that you are in the right directory
 
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab5.py
+
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
 
python3 ./CheckLab5.py -f -v lab5b
 
python3 ./CheckLab5.py -f -v lab5b
 
</source>
 
</source>
:::4. Before proceeding, make certain that you identify any and all errors in lab5b.py. When the checking script tells you everything is OK before proceeding to the next step.
+
::4. Before proceeding, make certain that you identify all errors in lab5b.py. When the checking script tells you everything is OK - proceed to the next step.
 
<br><br>
 
<br><br>
  
 
= INVESTIGATION 2: Exceptions and Error Handling =
 
= INVESTIGATION 2: Exceptions and Error Handling =
Running into errors in programming will be a common occurrence. In python when a error occurs, python raises a python object called an exception, which represents the error that occured. These exceptions are raised when python is no long able to handle what the code is trying to do. This section will give the programmer the ability to catch these exceptions when they happen and allow the program to continue running, however in many cases it might be a good idea to stop the program when an exception happens anyway.
+
 
 +
:Running into errors in programming will be a common occurrence. You should expect that it will happen for any code that you write. In python, when an error occurs, the python runtime raises an '''exception'''. This section will teach you to catch these exceptions when they happen and to allow the program to continue running, or to stop program execution with a readable error message.
  
 
== PART 1 - Handling Errors ==
 
== PART 1 - Handling Errors ==
There are a massive amount of exceptions, way too many to cover. But if you are searching for a specific exception check out the [https://docs.python.org/3/library/exceptions.html#exception-hierarchy Python Exception Documentation.]  
+
 
:'''Perform the Following Steps:'''
+
There is a <u>massive amount</u> of exceptions. Online references can be useful. If you are searching for a common exception check out the [https://docs.python.org/3/library/exceptions.html#exception-hierarchy Python Exception Documentation.]  
::#To start, open the ipython3 shell:<source lang="python">
+
 
ipython3
+
In this section, we will provide examples of how to handle a few exceptions when creating Python scripts.
</source>
+
 
::#Before we try and catch and detect exceptions, lets make sure we know a few ways to make errors in python. The following code will create an error:<source lang="python">
+
:#To start, open the ipython3 shell. Before attempting to handle exception errors, let's create an error, and then see how to can "handle" it:<br><source lang="python">
 
print('5' + 10)
 
print('5' + 10)
</source>
+
</source>You should get an exception error similar to the following:<source>
::#Immediately the following error occurs:<source lang="python">
 
 
---------------------------------------------------------------------------
 
---------------------------------------------------------------------------
TypeError                                Traceback (most recent call last)
+
Traceback (most recent call last)
<ipython-input-3-1b929b80ca50> in <module>()
+
Fiel "<stdin>", line 1, in <module>
----> 1 print('5' + 10)
+
TypeError: Can't convert 'int' object to str implicitly
  
TypeError: Can't convert 'int' object to str implicitly
+
</source><br>'''Question:''' According to the exception error message, what do you think caused the error?<br><br>
</source>
+
:#Click on the link '[https://docs.python.org/3/library/exceptions.html#concrete-exceptions https://docs.python.org/3/library/exceptions.html#concrete-exceptions.]' and  scroll or search for '''TypeError'''. Take a few moments to determine what a ''TypeError'' exception error means.<br><br>You should have learned that the TypeError exception error indicates a mismatch of a type (i.e. string, int, float, list, etc). If Python doesn't know how to handle it, perhaps we could change the number into a string or change the string into a number or at least provide a more user-friendly error message.<br><br>If we don't want the user of our program to have to learn how to read Python exceptions (which is a very good idea), we can catch/trap/handle this error when it happens. This is done with a specific block of code called a [https://docs.python.org/3/tutorial/errors.html#handling-exceptions '''try clause'''] where you place code in-between the '''try:''' and the '''except:''' coding blocks. In a general sense, it works like a modified if-else statement, where the try statement acts as a test, and the except statement will or will not handle the exception depending if it occurs or does NOT occur. That is to say, If no error occurs in the code contained in the '''except''' section, the script will continue as usual but if an error occurs in the except section, then it can be handled with additional coding (like an user-friendly error message).<br>Let's demonstrate to handle our TypeError error with code that first does not contain an error and then similar code that DOES generate an error.<br><br>
::#Firstly, what to look for here is the exception name '[https://docs.python.org/3/library/exceptions.html#TypeError TypeError.]' The Type error means a mismatch of a type(string, int, float, list, etc). Python doesn't know how to handle it, should it change the number into a string or change the string into a number?
+
:#The following code does NOT generate an error:<source lang="python">
::#If we want to write this program safely, we can catch this error while it's happening. This is done with a specific block of code called a [https://docs.python.org/3/tutorial/errors.html#handling-exceptions try clause] where you place code inbetween the '''try:''' and ''''except:'''. If no error occurs in the code the except portion will be skipped.
 
::#No Exception:<source lang="python">
 
 
try:
 
try:
 
     print(5 + 10)
 
     print(5 + 10)
 
except TypeError:
 
except TypeError:
     print('not an integer')
+
     print('At least one of the values is NOT an integer')
  
 
15
 
15
</source>
+
</source><br>You should notice that since there was NOT an error, the Python script performed the required task.<br><br>
::#TypeError Exception:<source lang="python">
+
:#The following code handles an exception error to provide user-friendly feedback that at least one of the values is not an integer:<source lang="python">
 
try:
 
try:
 
     print(5 + 'ten')
 
     print(5 + 'ten')
 
except TypeError:
 
except TypeError:
     print('not an integer')
+
     print('At least one of the values is NOT an integer')
  
not an integer
+
At least one of the values is NOT an integer
 
</source>
 
</source>
::#Try and open a file that doesn't exist:<source lang="python">
+
:#Let's generate another type of error where we try to open a file that doesn't exist:<source lang="python">
 
f = open('filethatdoesnotexist', 'r')
 
f = open('filethatdoesnotexist', 'r')
 
</source>
 
</source>
::#To catch this error specific error we could simply use:<source lang="python">
+
:#Now, catch and handle this exception error:<source lang="python">
 
try:
 
try:
 
     f = open('filethatdoesnotexist', 'r')
 
     f = open('filethatdoesnotexist', 'r')
Line 364: Line 383:
 
except FileNotFoundError:
 
except FileNotFoundError:
 
     print('no file found')
 
     print('no file found')
</source>
+
</source><br>Multiple exceptions can also be caught at the same time, such as does not exist, is a directory, or we don't have permission. <br><br>
::#Multiple exceptions can also be caught at the same time, such as does not exist, is a directory, or we don't have permission. Try removing permissions from the file, or creating a directory and opening it. <source lang="python">
+
:#To test out the error handling code (previously issued), try removing permissions from the file, or specify a directory instead of a regular file, and then try to open it:<source lang="python">
 
try:
 
try:
 
     f = open('filethatdoesnotexist', 'r')
 
     f = open('filethatdoesnotexist', 'r')
 
     f.write('hello world\n')
 
     f.write('hello world\n')
 
     f.close()
 
     f.close()
except (FileNotFoundError, PermissionError, IsADirectory):  
+
except (FileNotFoundError, PermissionError, IsADirectoryError):  
 
     print('failed to open file')
 
     print('failed to open file')
 
</source>
 
</source>
::#If you spend some time looking at the [https://docs.python.org/3/library/exceptions.html#exception-hierarchy Python Exception Hierarchy] we can see exactly how errors get caught in python. FileNotFoundError, PermissionError, and IsADirectory all inherit from OSError, this means that while using more specific errors might be useful for better error messages and handling, it's not always possible to catch every error all the time.::#Another way to catch multiple exceptions:<source lang="python">
+
:#By taking the time to view the [https://docs.python.org/3/library/exceptions.html#exception-hierarchy Python Exception Hierarchy], you can see how errors get caught in python. The options '''FileNotFoundError''', '''PermissionError''', and '''IsADirectory''' are all inherited from '''OSError'''. This means that while using more specific errors might be useful for better error messages and handling, it's not always possible to catch every error all the time.
 +
:#Another way to catch multiple exceptions is with separate <code>except</code> bloks:<source lang="python">
 
try:
 
try:
 
     f = open(abc, 'r')
 
     f = open(abc, 'r')
Line 387: Line 407:
 
     print('unknown error occured')
 
     print('unknown error occured')
 
     raise
 
     raise
</source>
+
</source>When catching multiple exceptions, make certain to catch the <u>lowest</u> ones contained in the exception-hierarchy first. For example, if you put 'Exception' first, both 'OSError' and 'FileNotFoundError', would never get caught.<br><br>'''TIP:''' In python it's usually best to 'try:' and 'except:' code rather than to attempt to anticipate everything that could go wrong with '''if''' statements. For example, instead of checking to see if a file exists and we have read permissions, it can be better to just try and read the file and fail and catch any errors with 'OSError'.
::#When catching multiple exceptions, make sure to catch the lowest ones on the hierarchy first. If you put 'Exception' first, both 'OSError' and 'FileNotFoundError', would never get caught.
 
::#In python it's usually best to 'try:' and 'except:' code rather than to try and determine everything that could go wrong with logic and if statements. For example, instead of checking to see if a file exists and we have read permissions, it can be better to just try and read the file and fail and catch any errors with 'OSError'.
 
  
 
=== Create a Python Script Which Handles Errors ===
 
=== Create a Python Script Which Handles Errors ===
:'''Perform the Following Instructions'''
+
:#Create the '''~/ops435/lab5/lab5c.py''' script.
::#Create the '''~/ops435/lab5/lab5c.py''' script.
+
:#Use the following as a template:<source lang="python">
::#Use the following as a template:<source lang="python">
 
 
#!/usr/bin/env python3
 
#!/usr/bin/env python3
  
Line 413: Line 430:
  
  
::::'''Sample Run 1:'''<source lang="python">
+
:::'''Sample Run 1:'''<source>
 
python3 lab5c.py  
 
python3 lab5c.py  
 
15
 
15
Line 422: Line 439:
 
</source>
 
</source>
  
::::'''Sample Run 2 (with import):'''<source lang="python">
+
:::'''Sample Run 2 (with import):'''<source>
 
import lab5c                 
 
import lab5c                 
  
Line 450: Line 467:
  
  
:::3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.<source lang="bash">
+
::3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.<source lang="bash">
 
cd ~/ops435/lab5/
 
cd ~/ops435/lab5/
 
pwd #confirm that you are in the right directory
 
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab5.py
+
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
 
python3 ./CheckLab5.py -f -v lab5c
 
python3 ./CheckLab5.py -f -v lab5c
 
</source>
 
</source>
:::4. Before proceeding, make certain that you identify any and all errors in lab5c.py. When the checking script tells you everything is OK before proceeding to the next step.
+
::4. Before proceeding, make certain that you identify any and all errors in lab5c.py. When the checking script tells you everything is OK before proceeding to the next step.
 
<br><br>
 
<br><br>
  
Line 462: Line 479:
 
:'''Have Ready to Show Your Instructor:'''
 
:'''Have Ready to Show Your Instructor:'''
  
::<span style="color:green;font-size:1.5em;">&#x2713;</span> x
+
::<span style="color:green;font-size:1.5em;">&#x2713;</span> Output of: <code>./CheckLab5.py -f -v</code>
::<span style="color:green;font-size:1.5em;">&#x2713;</span> x
+
::<span style="color:green;font-size:1.5em;">&#x2713;</span> Output of: <code>cat lab5a.py lab5b.py lab5c.py</code>
::<span style="color:green;font-size:1.5em;">&#x2713;</span> Lab4 logbook notes completed
+
 
= Practice For Quizzes, Tests, Midterm & Final Exam =
+
= LAB REVIEW =
# x
+
 
# x
+
# What is the purpose of a '''file object'''?
# x
+
# Write a Python command to '''open''' the text file called '''customers.txt''' for read-only operations.
 +
# Write Python code to efficiently store the contents of the file in question #2 as a large string (including new-line characters) called '''customer-data'''.
 +
# Write Python code to store the contents of the file in question #2 as a list, removing the new-line characters.
 +
# What is the purpose of '''closing''' an open file? Write a Python command to close the file opened in question #2.
 +
# Write the Python command to confirm you successfully closed the '''customers.txt''' file in question #5. What is the returned status from that command to indicate that the file has been closed?
 +
# What is the difference between opening a file for '''writing''' data as opposed to opening a file for '''appending''' data? What can be the consequence if you don't understand the difference between writing and appending data?
 +
# Write a Python command to open the file '''customer-data.txt''' for writing data.
 +
# Write a Python command to save the text: '''customer 1: Guido van Rossum''' (including a new-line character) to the opened file called '''customer-data.txt'''
 +
# Briefly explain the process writing a list as separate lines to an open file.
 +
# What is the purpose of '''handling''' exception errors?
 +
# Write a Python script to prompt a user for the name of the file to open. Use exception error handling to provide an error message that the specific file name (display that exact name) does not exist; otherwise, open the file for reading and display the entire contents of the file (line-by-line).
 +
 
 +
[[Category:OPS435-Python]]

Latest revision as of 09:26, 21 January 2020

** DO NOT USE - TO BE UPDATED FOR CENTOS 8.0 **

LAB OBJECTIVES

So far, you have created Python scripts to prompt a user to input data from the keyboard. When creating Python scripts, you may also need to be able to process large volumes of information, or store processed data for further processing. The first investigation in this lab will focus on file management, opening files, saving data to files, and reading files.
NOTE: Since many tasks that system administrators perform deal with files, this is a crucial skill to understand.
It is very important to provide logic in your Python script in case it encounters an error. An example would be an invalid path-name or trying to close a file that is already closed. The second investigation in this lab will look into how the Python interpreter handle errors (commonly referred to as "exception handling") at run time, and learn how to write Python code that will run gracefully even when problems occur during program execution.


PYTHON REFERENCE

In previous labs, you have been advised to make notes and use online references. This also relates to working with files and learning about objected oriented programming. You may be "overwhelmed" with the volume of information involved in this lab.
Below is a table with links to useful online Python reference sites (by category). You may find these references useful when performing assignments, etc.
Category Resource Link
Handling Errors & Exceptions
Errors & Exceptions
Built-in Exceptions
Built-in Exceptions
Data Manipulation via Files
Reading & Writing Files
Opening Files
Open File Usage
Using Classes
Classes

INVESTIGATION 1: Working with Files

You will now learn how to write Python scripts in order to open text files, to read the contents within a text file, to process the contents, and finally to write the processed contents back into a file. These operations are very common, and are used extensively in programming. Examples of file operations would include situations such as logging output, logging errors, reading and creating configuration/temporary files, etc.
Files are accessed through the use of file objects. An object is a storage location which stores data in the form of attributes (variables) and methods (functions). Creating our own objects will be covered later in investigation 3.

PART 1 - Reading Data From Files

Perform the Following Steps:
  1. Create a new python file for testing.
  2. Create a new text file in the lab5 directory:
    cd ~/ops435/lab5
    vim ~/ops435/lab5/data.txt
  3. Place the following content inside the new text file and save it:
    Hello World
    This is the second line
    Third line
    Last line

    In order to read data from a text file, we need to create an object that will be used to access the data in a file. In some programming languages (like C) this is called a file descriptor, or a file pointer. In Python, it's an object.

  4. Now lets write some python code to open this created file for reading. We will define and object called "f" in order to help retrieve content from our text file. Issue the following:
    f = open('data.txt', 'r')

    The open() function takes two string arguments: a path to a file, and a mode option (to ask for reading, writing, appending, etc). The open() function will return a file object to us, this file object will allow us to read the lines inside the file.

  5. Here are the most useful functions for text file manipulation:
    f.read()            # read all lines and stores in a string
    f.readlines()       # read all lines and stores in a list
    f.readline()        # read first line, if run a second time it will read the second line, then third
    f.close()           # close the opened file
  6. Next, read data from the buffer of the opened file and store the contents into a variable called read_data, and then confirm the contents of the variable read_data:
    read_data = f.read()
    print(read_data)

    After you have completed accessing data within a file, you should close the file in order to free up the computer resources. It is sometimes useful to first confirm that the file is still open prior to closing it. But really you should know - it's your code that would have opened it:
    f.close()                               # This method will close the file

    Let's take a moment to revisit the file read sequence. The following code sequence will open a file, store the contents of a file into a variable, close the file and provide confirmation that the file has been closed:
    f = open('data.txt', 'r')               # Open file
    read_data = f.read()                    # Read from file
    f.close()                               # Close file
  7. read_data in this case contains the data from the file in a single long string. The end of each line in the file will show the special character '\n' which represents the newline character in a file used to separate lines (or records in a traditional "flat database file"). It would be convenient to split the line on the new-line characters, so each line can be stored as an item in a list.

  8. Store the contents of our file into a list called list_of_lines:
    read_data.split('\n')                       # Returns a list
    list_of_lines = read_data.split('\n')       # Saves returned list in variable
    print(list_of_lines)

    Although the above sequence works, there are functions and methods the we can use with our object (called "f") to place lines from our file into a list. This would help to reduce code and is considered a more common method to store multiple lines or records within a list.

  9. Try these two different means to store data into a list more efficiently:
    # METHOD 1:
    
    f = open('data.txt', 'r')
    method1 = list(f)
    f.close()
    print(method1)
    
    # METHOD 2:
    
    f = open('data.txt', 'r')
    method2 = f.readlines()
    f.close()
    print(method2)

Create a Python Script Demonstrating Reading Files

  1. Create the ~/ops435/lab5/lab5a.py script.
  2. Use the following as a template:
    #!/usr/bin/env python3
    
    def read_file_string(file_name):
        # Takes a filename string, returns a string of all lines in the file
    
    def read_file_list(file_name):
        # Takes a filename string, returns a list of lines without new-line characters
    
    if __name__ == '__main__':
        file_name = 'data.txt'
        print(read_file_string(file_name))
        print(read_file_list(file_name))
  • This Python script will read the same file (data.txt) that you previously created
  • The read_file_string() function should return a string
  • The read_file_list() function should return a list
  • The read_file_list() function must remove the new-line characters from each line in the list
  • Both functions must accept one argument which is a string
  • The script should show the exact output as the samples
  • The script should contain no errors
Sample Run 1:
python3 lab5a.py 
Hello World
This is the second line
Third line
Last line
['Hello World', 'This is the second line', 'Third line', 'Last line']
Sample Run 2 (with import):
import lab5a        
file_name = 'data.txt'                             
print(lab5a.read_file_string(file_name))
# Will print 'Hello World\nThis is the second line\nThird line\nLast line\n'
print(lab5a.read_file_list(file_name))
# Will print ['Hello World', 'This is the second line', 'Third line', 'Last line']
3. Download the checking script and check your work. Enter the following commands from the bash shell.
cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5a
4. Before proceeding, make certain that you identify all errors in lab5a.py. When the checking script tells you everything is OK - proceed to the next step.



PART 2 - Writing To Files

Up to this point, you have learned how to access text from a file. In this section, you will learn how to write text to a file. Writing data to a file is useful for creating new content in a file or updating (modifying) existing data contained within a file.
Stop (medium size).png
Risk of Losing File Contents
A common problem that new Python programmers may encounter is to accidentally erase existing contents of a file when writing new data to a file. When opening files for writing (using the 'w' open function option), Python assumes existing content in the file is no longer wanted and it's immediately deleted; therefore, if you wish to write data to a file but keep existing content, you need to use the open file option 'a' (append new data to a file).
  1. When opening a file for writing, the 'w' option is specified with the open() function. When the 'w' option is specified - previous (existing) content inside the file is deleted. This deletion takes place the moment the open() function is executed, not when writing to the file. If the file that is being written to doesn't exist, the file will be created upon the file opening process.

  2. Create a temporary Python file and open a non-existent data file (called file1.txt) for writing:
    f = open('file1.txt', 'w')
  3. To confirm that the new file now exists and is empty, issue the following shell command:
    ls -l file1.txt
    To add lines of text to the file, you can use the write() method for the file object. Typically you end every line in a text file with the special character '\n' to represent a "new line". Multiple lines may also be placed inside a single write operation: simply put the special character '\n' wherever a line should end.

  4. Try adding multiple lines:
    f.write('Line 1\nLine 2 is a little longer\nLine 3 is too\n')
    Once the write() method has completed, the final step is to close() the file. The file MUST be closed properly or else data will not consistently be written to the file. NOTE: Not closing a file can lead to corrupted or missing file contents.:
    f.close()
  5. View the contents of the file in the shell to make sure the data was written successfully:
    cat file1.txt
    You will now create a new file called file2.txt, but this time run multiple write() methods in sequence. You will often write to a file multiple times inside a loop:
    f = open('file2.txt', 'w')
    f.write('Line 1\nLine 2 is a little longer\nLine 3 is as well\n')
    f.write('This is the 4th line\n')
    f.write('Last line in file\n')
    f.close()
  6. Issue the following shell command to confirm that the contents were written to file2.txt:
    cat file2.txt
Important.png
Make Backup Copies of Your Data Files
Since you might make a mistake, and accidentally destroy file contents when writing to your file, it is highly recommended to make backup copies of your files prior to running your Python scripts. This can be particularly useful when performing any future assignment involving writing to files.
Stop (medium size).png
Make Backup Copies of ALL your files
Since you are now writing code that opens files for writing you may accidentally truncate the wrong file (like your assignment file, for example). Make regular backups of all your work. Just copy it to a USB stick or if you're bored - learn version control.

  1. Issue the following shell commands to backup both of your newly-created files and confirm backup:
    cp file1.txt file1.txt.bk
    cp file2.txt file2.txt.bk
    ls -l file*
  2. Let's demonstrate what can happen if you perform an incorrect write() operation:
    f = open('file2.txt', 'w')
    cat file2.txt
    You should notice that the previous content in your file2.txt file was destroyed. Why do you you think the previous data is no longer there?
  3. Restore your file from the backup and verify the backup restoration:
    cp file2.txt.bk file2.txt
    cat file2.txt
    To avoid overwriting the contents of a file, we can append data to the end of the file instead. Use the option 'a' instead of 'w' to perform appending:

    f = open('file1.txt', 'a')
    f.write('This is the 4th line\n')
    f.write('Last line in file\n')
    f.close()
    cat file1.txt
    The final thing to consider when writing to files is to make certain that the values being written are strings. This means that before trying to place integers, floats, lists, or dictionaries into a file, first either convert the value using str() function or extract the specific strings from items in the list.

  4. In this example we convert a single number and all the numbers in a list to strings before writing them to a file:
    my_number = 1000
    my_list = [1,2,3,4,5]
    f = open('file3.txt', 'w')
    f.write(str(my_number) + '\n')
    for num in my_list:
        f.write(str(num) + '\n')
    f.close()
  5. Confirm that the write() operation was successful
    cat file3.txt

Create a Python Script Demonstrating Writing to Files

  1. Copy ~/ops435/lab5/lab5a.py script to ~/ops435/lab5/lab5b.py script (We need the previous read functions that you created).
  2. Add the following functions below the two functions that you already created:
    def append_file_string(file_name, string_of_lines):
        # Takes two strings, appends the string to the end of the file
    
    def write_file_list(file_name, list_of_lines):
        # Takes a string and list, writes all items from list to file where each item is one line
    
    def copy_file_add_line_numbers(file_name_read, file_name_write):
        # Takes two strings, reads data from first file, writes data to new file, adds line number to new file
  3. Replace the main section of your Python script near the bottom with the following:
    if __name__ == '__main__':
        file1 = 'seneca1.txt'
        file2 = 'seneca2.txt'
        file3 = 'seneca3.txt'
        string1 = 'First Line\nSecond Line\nThird Line\n'
        list1 = ['Line 1', 'Line 2', 'Line 3']
        append_file_string(file1, string1)
        print(read_file_string(file1))
        write_file_list(file2, list1)
        print(read_file_string(file2))
        copy_file_add_line_numbers(file2, file3)
        print(read_file_string(file3))
append_file_string():
  1. Takes two string arguments
  2. Appends to the file(Argument 1) all data from the string(Argument 2)
write_file_list():
  1. Takes two arguments: a string and a list
  2. Writes to file(Argument 1) all lines of data found in the list(Argument 2)
copy_file_add_line_numbers():
  1. Takes two arguments: Both are files path-names (which happen to be strings)
  2. Reads all data from first file(Argument 1), and writes all lines into second file(Argument 2) adding line numbers
  3. Line numbers should be added to the beginning of each line with a colon next to them(see sample output below for reference)
  4. Hint: Use an extra variable for the line number
Sample Run 1:
rm seneca1.txt seneca2.txt seneca3.txt
./lab5b.py 
First Line
Second Line
Third Line

Line 1
Line 2
Line 3

1:Line 1
2:Line 2
3:Line 3
Sample Run 2 (run second time):
python3 lab5b.py 
First Line
Second Line
Third Line
First Line
Second Line
Third Line

Line 1
Line 2
Line 3

1:Line 1
2:Line 2
3:Line 3
Sample Run 3 (with import):
import lab5b
file1 = 'seneca1.txt'
file2 = 'seneca2.txt'
file3 = 'seneca3.txt'
string1 = 'First Line\nSecond Line\nThird Line\n'
list1 = ['Line 1', 'Line 2', 'Line 3']

lab5b.append_file_string(file1, string1)

lab5b.read_file_string(file1)
# Will print 'First Line\nSecond Line\nThird Line\nFirst Line\nSecond Line\nThird Line\n'

lab5b.write_file_list(file2, list1)

lab5b.read_file_string(file2)
# Will print 'Line 1\nLine 2\nLine 3\n'

lab5b.copy_file_add_line_numbers(file2, file3)

lab5b.read_file_string(file3)
# Will print '1:Line 1\n2:Line 2\n3:Line 3\n'
3. Download the checking script and check your work. Enter the following commands from the bash shell.
cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5b
4. Before proceeding, make certain that you identify all errors in lab5b.py. When the checking script tells you everything is OK - proceed to the next step.



INVESTIGATION 2: Exceptions and Error Handling

Running into errors in programming will be a common occurrence. You should expect that it will happen for any code that you write. In python, when an error occurs, the python runtime raises an exception. This section will teach you to catch these exceptions when they happen and to allow the program to continue running, or to stop program execution with a readable error message.

PART 1 - Handling Errors

There is a massive amount of exceptions. Online references can be useful. If you are searching for a common exception check out the Python Exception Documentation.

In this section, we will provide examples of how to handle a few exceptions when creating Python scripts.

  1. To start, open the ipython3 shell. Before attempting to handle exception errors, let's create an error, and then see how to can "handle" it:
    print('5' + 10)
    You should get an exception error similar to the following:
    ---------------------------------------------------------------------------
    Traceback (most recent call last)
     Fiel "<stdin>", line 1, in <module>
    TypeError: Can't convert 'int' object to str implicitly

    Question: According to the exception error message, what do you think caused the error?

  2. Click on the link 'https://docs.python.org/3/library/exceptions.html#concrete-exceptions.' and scroll or search for TypeError. Take a few moments to determine what a TypeError exception error means.

    You should have learned that the TypeError exception error indicates a mismatch of a type (i.e. string, int, float, list, etc). If Python doesn't know how to handle it, perhaps we could change the number into a string or change the string into a number or at least provide a more user-friendly error message.

    If we don't want the user of our program to have to learn how to read Python exceptions (which is a very good idea), we can catch/trap/handle this error when it happens. This is done with a specific block of code called a try clause where you place code in-between the try: and the except: coding blocks. In a general sense, it works like a modified if-else statement, where the try statement acts as a test, and the except statement will or will not handle the exception depending if it occurs or does NOT occur. That is to say, If no error occurs in the code contained in the except section, the script will continue as usual but if an error occurs in the except section, then it can be handled with additional coding (like an user-friendly error message).
    Let's demonstrate to handle our TypeError error with code that first does not contain an error and then similar code that DOES generate an error.

  3. The following code does NOT generate an error:
    try:
        print(5 + 10)
    except TypeError:
        print('At least one of the values is NOT an integer')
    
    15

    You should notice that since there was NOT an error, the Python script performed the required task.

  4. The following code handles an exception error to provide user-friendly feedback that at least one of the values is not an integer:
    try:
        print(5 + 'ten')
    except TypeError:
        print('At least one of the values is NOT an integer')
    
    At least one of the values is NOT an integer
  5. Let's generate another type of error where we try to open a file that doesn't exist:
    f = open('filethatdoesnotexist', 'r')
  6. Now, catch and handle this exception error:
    try:
        f = open('filethatdoesnotexist', 'r')
        f.write('hello world\n')
        f.close()
    except FileNotFoundError:
        print('no file found')

    Multiple exceptions can also be caught at the same time, such as does not exist, is a directory, or we don't have permission.

  7. To test out the error handling code (previously issued), try removing permissions from the file, or specify a directory instead of a regular file, and then try to open it:
    try:
        f = open('filethatdoesnotexist', 'r')
        f.write('hello world\n')
        f.close()
    except (FileNotFoundError, PermissionError, IsADirectoryError): 
        print('failed to open file')
  8. By taking the time to view the Python Exception Hierarchy, you can see how errors get caught in python. The options FileNotFoundError, PermissionError, and IsADirectory are all inherited from OSError. This means that while using more specific errors might be useful for better error messages and handling, it's not always possible to catch every error all the time.
  9. Another way to catch multiple exceptions is with separate except bloks:
    try:
        f = open(abc, 'r')
        f.write('hello world\n')
        f.close()
    except (FileNotFoundError, PermissionError): 
        print('file does not exist or wrong permissions')
    except IsADirectoryError:
        print('file is a directory')
    except OSError:
        print('unable to open file')
    except:
        print('unknown error occured')
        raise
    When catching multiple exceptions, make certain to catch the lowest ones contained in the exception-hierarchy first. For example, if you put 'Exception' first, both 'OSError' and 'FileNotFoundError', would never get caught.

    TIP: In python it's usually best to 'try:' and 'except:' code rather than to attempt to anticipate everything that could go wrong with if statements. For example, instead of checking to see if a file exists and we have read permissions, it can be better to just try and read the file and fail and catch any errors with 'OSError'.

Create a Python Script Which Handles Errors

  1. Create the ~/ops435/lab5/lab5c.py script.
  2. Use the following as a template:
    #!/usr/bin/env python3
    
    def add(number1, number2):
        # Add two numbers together, return the result, if error return string 'error: could not add numbers'
    
    def read_file(filename):
        # Read a file, return a list of all lines, if error return string 'error: could not read file'
    
    if __name__ == '__main__':
        print(add(10,5))                        # works
        print(add('10',5))                      # works
        print(add('abc',5))                     # exception
        print(read_file('seneca2.txt'))         # works
        print(read_file('file10000.txt'))       # exception


Sample Run 1:
python3 lab5c.py 
15
15
error: could not add numbers
['Line 1\n', 'Line 2\n', 'Line 3\n']
error: could not read file
Sample Run 2 (with import):
import lab5c                

lab5c.add(10,5)             
15                          

lab5c.add('10',5)           
15                          

lab5c.add('10','5')         
15                          

lab5c.add('abc','5')                                                                                                                     
'error: could not add numbers'                                  

lab5c.add('hello','world')                                                                                                               
'error: could not add numbers'                                  

lab5c.read_file('seneca2.txt')
['Line 1\n', 'Line 2\n', 'Line 3\n']           

lab5c.read_file('file10000.txt')   
error: could not read file'


3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.
cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5c
4. Before proceeding, make certain that you identify any and all errors in lab5c.py. When the checking script tells you everything is OK before proceeding to the next step.



LAB 5 SIGN-OFF (SHOW INSTRUCTOR)

Have Ready to Show Your Instructor:
Output of: ./CheckLab5.py -f -v
Output of: cat lab5a.py lab5b.py lab5c.py

LAB REVIEW

  1. What is the purpose of a file object?
  2. Write a Python command to open the text file called customers.txt for read-only operations.
  3. Write Python code to efficiently store the contents of the file in question #2 as a large string (including new-line characters) called customer-data.
  4. Write Python code to store the contents of the file in question #2 as a list, removing the new-line characters.
  5. What is the purpose of closing an open file? Write a Python command to close the file opened in question #2.
  6. Write the Python command to confirm you successfully closed the customers.txt file in question #5. What is the returned status from that command to indicate that the file has been closed?
  7. What is the difference between opening a file for writing data as opposed to opening a file for appending data? What can be the consequence if you don't understand the difference between writing and appending data?
  8. Write a Python command to open the file customer-data.txt for writing data.
  9. Write a Python command to save the text: customer 1: Guido van Rossum (including a new-line character) to the opened file called customer-data.txt
  10. Briefly explain the process writing a list as separate lines to an open file.
  11. What is the purpose of handling exception errors?
  12. Write a Python script to prompt a user for the name of the file to open. Use exception error handling to provide an error message that the specific file name (display that exact name) does not exist; otherwise, open the file for reading and display the entire contents of the file (line-by-line).