Difference between revisions of "OPS435 Python3 Assignment 2"

From CDOT Wiki
Jump to: navigation, search
(Assignment 2 - Usage Report)
(Assignment 2 - Usage Report)
Line 6: Line 6:
 
'''What you need''' A github account with a private repository named ops435-a2
 
'''What you need''' A github account with a private repository named ops435-a2
  
'''Due Date:''' Please follow the four stages of submission schedule:
+
'''Due Date:''' Please follow the three stages of submission schedule:
* Complete the user requirement document by November 15, 2019 on github,
+
* Complete the detail algorithm for this assignment script by March 20, 2020 on github,
* Complete the your Python script by November 22, 2019 on github
+
* Complete the your Python script coding and upload to your vm in myvmlab by March 26, 2020 on github
* Complete the testing and debugging by November 29, 2019 on github, and also submit the user requirement document file, test results and the python script to blackboard.
+
* Complete the testing and debugging by April 3 on the vm in myvmlab, and also submit the detail algorithm file (algorithm.txt), test results and the python script (a1_[seneca_id].py)to blackboard.
  
 
'''Late Penalty:''' 20% per school day, and note that this assignment must be completed satisfactorily in order to pass the course even if you get zero mark for this assignment.
 
'''Late Penalty:''' 20% per school day, and note that this assignment must be completed satisfactorily in order to pass the course even if you get zero mark for this assignment.
Line 27: Line 27:
 
asmith  pts/11      10.40.105.130    Tue Feb 13 14:07:43 2018 - Tue Feb 13 16:07:43 2018  (02:00)
 
asmith  pts/11      10.40.105.130    Tue Feb 13 14:07:43 2018 - Tue Feb 13 16:07:43 2018  (02:00)
 
</pre>
 
</pre>
It is always desirable to have a daily or weekly usage reports by user or by remote host based on the above information.
+
It is always desirable to have a daily, weekly, or monthly usage reports by user or by remote host based on the above information.
  
 
== Tasks for this assignment ==
 
== Tasks for this assignment ==
 
In this assignment, your should preform the following activities:
 
In this assignment, your should preform the following activities:
# Complete an User Requirement Document for producing daily and weekly usage reports by user or by remote host based on the information stored in any given files generated from the 'last' command.  
+
# Complete a detail algorithm for producing daily, weekly, and monthly usage reports by user or by remote host based on the information stored in any given files generated from the 'last' command, either from a text file or real-time.  
# Once you have complete the User Requirement Document, you should then <b>design the structure of your python script</b> by identifying the appropriate python objects, functions and modules to be used for each task in your algorithm and the main control logic. Make sure to identify the followings:
+
# Once you have complete the detail algorithm, you should then <b>design the structure of your python script</b> by identifying the appropriate python objects, functions and modules to be used for each task in your algorithm and the main control logic. Make sure to identify the followings:
 
## input data,  
 
## input data,  
 
## computation tasks, and  
 
## computation tasks, and  
 
## outputs.
 
## outputs.
# implement your computational solution using a single python script. You can use any built-in functions and functions from the allowed python modules to implement your solution.  
+
# implement your computational solution using a single python script. You can use any built-in functions and functions from the python modules list in the "Allowed Python Modules" section below to implement your solution.  
 
# Test and review your working python code to see whether you can improve the interface of each function to facilitate better code re-use (this process is called <b>refactoring</b>).
 
# Test and review your working python code to see whether you can improve the interface of each function to facilitate better code re-use (this process is called <b>refactoring</b>).
  
== Allowed Python Modules for this assignment ==
+
== Allowed Python Modules ==
 
* the <b>os, sys</b> modules
 
* the <b>os, sys</b> modules
 
* the <b>argparse</b> module
 
* the <b>argparse</b> module
* The <b>time</b> module or the <b>a2</b> module
+
* The <b>time</b> module  
 +
* The <b>ur_funcs module (provide by your teacher on github)
 
** [https://docs.python.org/3/howto/argparse.html Argparse Tutorial] - should read this first.
 
** [https://docs.python.org/3/howto/argparse.html Argparse Tutorial] - should read this first.
 
** [https://docs.python.org/3/library/argparse.html Argparse API reference information page]
 
** [https://docs.python.org/3/library/argparse.html Argparse API reference information page]
* the <b>time</b> module or the <b>a2</b>module, <b>but not both</b>
 
** [https://www.tutorialspoint.com/python3/python_date_time.htm Time module Tutorial]
 
** [https://docs.python.org/3/library/time.html Time module API reference page]
 
  
 
== Instructions ==
 
== Instructions ==
Line 76: Line 74:
 
   -v, --verbose        tune on output verbosity
 
   -v, --verbose        tune on output verbosity
  
Copyright 2019 - Raymond Chan  
+
Copyright 2020 - Raymond Chan  
 
</pre>
 
</pre>
 
Replace the last line with your own full name  
 
Replace the last line with your own full name  
  
  
If there is only one file name provided at the command line, read the login/logout records from the contents of the given file. The format of each line in the file should be the same as the output of 'last -Fiw'. Filter out incomplete login/logout record (hints: check for the number of fields in each record).
+
If there is only one file name provided at the command line, read the login/logout records from the contents of the given file. If the file name is "online", get the record on the system your script is being execute using the Linux command "last -iwF". The format of each line in the file should be the same as the output of 'last -Fiw'. Filter out incomplete login/logout record (hints: check for the number of fields in each record).
  
 
If there is more than one file name provided, merge all the files together with the first one at the top and the last one at the bottom. Read and process the file contents in that order in your program.
 
If there is more than one file name provided, merge all the files together with the first one at the top and the last one at the bottom. Read and process the file contents in that order in your program.
Line 89: Line 87:
 
All your Python codes for this assignment must be placed in a <font color='red'><b><u>single source file</u></b></font>. Please include the following declaration by <b><u>you</u></b> as the <font color='blue'><b>script level docstring</b></font> in your Python source code file (replace [Student_id] with your Seneca email user name, and "Student Name" with your own name):
 
All your Python codes for this assignment must be placed in a <font color='red'><b><u>single source file</u></b></font>. Please include the following declaration by <b><u>you</u></b> as the <font color='blue'><b>script level docstring</b></font> in your Python source code file (replace [Student_id] with your Seneca email user name, and "Student Name" with your own name):
  
<source>OPS435 Assignment 2 - Fall 2019
+
<source>OPS435 Assignment 2 - Winter 2020
Program: ur_[Stduent_id].py
+
Program: <b>ur_[seneca_id].py</b>
Author: "Student Name"
+
Author: "<font color='red'>Student Name</font>"
The python code in this file ur_[Student_id].py is original work written by
+
The python code in this file <b>ur_[seneca_id].py</b> is original work written by
"Student Name". No code in this file is copied from any other source  
+
"<font color='red'>Student Name</font>". No code in this file is copied from any other source  
 
including any person, textbook, or on-line resource except those provided
 
including any person, textbook, or on-line resource except those provided
 
by the course instructor. I have not shared this python file with anyone
 
by the course instructor. I have not shared this python file with anyone
Line 219: Line 217:
 
</pre>
 
</pre>
  
== Suggested workflow for this assignment ==
+
== Suggested work-flow for this assignment ==
 
=== create a <font color="blue">private</font> repository on github ===
 
=== create a <font color="blue">private</font> repository on github ===
 
* Use the same github account you used for your assignment 1 repository.
 
* Use the same github account you used for your assignment 1 repository.
Line 229: Line 227:
 
* <b><font color='blue'>Make sure that your professor accepted your invitation from github.com.</font></b>
 
* <b><font color='blue'>Make sure that your professor accepted your invitation from github.com.</font></b>
  
=== User Requirement Document ===
+
=== Detail Algorithm Document ===
Follow the standard computation procedure: input - process - ouput when creating the user requirement document for this assignment.
+
Follow the standard computation procedure: input - process - ouput when creating the algorithm document for this assignment.
 
==== input ====
 
==== input ====
* get data (command line arguments/options) from the user using your script
+
* get data (command line arguments/options) from the user using the functions provided by the argparse module
 
* according to the arguments/options given at the command line, take appropriate processing action.  
 
* according to the arguments/options given at the command line, take appropriate processing action.  
 
==== processing ====
 
==== processing ====
Line 249: Line 247:
  
 
To  help you with this assignment, you can use the ur_template.py in the sample ops435-a2 repository as a starting point in designing your own Python Usage Report script.
 
To  help you with this assignment, you can use the ur_template.py in the sample ops435-a2 repository as a starting point in designing your own Python Usage Report script.
 +
<font color='blue'><b>If you don't have enough time to create all the functions for the data processing steps, you should study the functions in the ur_funcs.py (provided by your teacher), pick and use the one that may help. If you use any of the functions from ur_funcs.py, there will be a cost of 10% to your overall grade. If you create all the functions yourself, you will get a bonus of 10%.</b></font>
  
 
=== Python script coding and debugging ===
 
=== Python script coding and debugging ===
Line 254: Line 253:
 
Once you have finished coding a function, you should start a Python3 interactive shell, import your functions and manually test each function and verify its correctness.
 
Once you have finished coding a function, you should start a Python3 interactive shell, import your functions and manually test each function and verify its correctness.
 
=== Final Test===
 
=== Final Test===
Once you have all the individual function tested and that each is working properly, perform the final test with test data provided by your professor and verify that your script produces the correct results before submitting your python program on Blackboard.
+
Once you have all the individual function tested and that each is working properly, perform the final test with test data provided by your professor and verify that your script produces the correct results before submitting your python program on Blackboard. Upload all the files for this assignment 2 to your vm in myvmlab and perform the final test.
  
 
== Sample login/logout records file and sample test run results==
 
== Sample login/logout records file and sample test run results==
Line 286: Line 285:
  
 
== Submission ==
 
== Submission ==
* Stage 1: upload your user requirement document file to your ops435-a2 repository in github.com by November 15, 2019
+
* Stage 1: upload your algorithm document file to your ops435-a2 repository in github.com by March 20, 2020
* Stage 2: upload your python script for this assignment to your ops435-a2 repository in github.com by November 22, 2019
+
* Stage 2: upload your python script for this assignment to your ops435-a2 repository in github.com and to your vm in myvmlab by March 26, 2020
* Stage 3: After fully tested and debugged your python script for this assignment, update your user requirement document, your python script, and your test results to your ops435-a2 repository in github.com by November 29, 2019. Also submit the user requirement document, and the python script to blackboard by November 29, 2019
+
* Stage 3: After fully tested and debugged your python script for this assignment, update your algorithm, your python script, and your est results to your ops435-a2 repository in github.com. Also submit the algorithm document, the python script and final test result to blackboard by April 3, 2020

Revision as of 01:14, 11 March 2020


Assignment 2 - Usage Report

Weight: 15% of the overall grade

What you need A github account with a private repository named ops435-a2

Due Date: Please follow the three stages of submission schedule:

  • Complete the detail algorithm for this assignment script by March 20, 2020 on github,
  • Complete the your Python script coding and upload to your vm in myvmlab by March 26, 2020 on github
  • Complete the testing and debugging by April 3 on the vm in myvmlab, and also submit the detail algorithm file (algorithm.txt), test results and the python script (a1_[seneca_id].py)to blackboard.

Late Penalty: 20% per school day, and note that this assignment must be completed satisfactorily in order to pass the course even if you get zero mark for this assignment.

Overview

Most system administrators would like to know the utilization of their systems by their users. On a Linux system, each user's login records are normally stored in the binary file /var/log/wtmp. The login records in this binary file can not be viewed or edited directly using normal Linux text commands like 'less', 'cat', etc. The 'last' command is often used to display the login records stored in this file in a human readable form. Please check the man page of the 'last' command for available options. The following is the contents of the file named "usage_data_file", which is a sample output of the 'last' command with the '-Fiw' flag on:

$ last -Fiw > usage_data_file
$ cat usage_data_file
rchan    pts/9        10.40.91.236     Tue Feb 13 16:53:42 2018 - Tue Feb 13 16:57:02 2018  (00:03)    
cwsmith  pts/10       10.40.105.130    Wed Feb 14 23:09:12 2018 - Thu Feb 15 02:11:23 2018  (03:02)
rchan    pts/2        10.40.91.236     Tue Feb 13 16:22:00 2018 - Tue Feb 13 16:45:00 2018  (00:23)    
rchan    pts/5        10.40.91.236     Tue Feb 15 16:22:00 2018 - Tue Feb 15 16:55:00 2018  (00:33)    
asmith   pts/2        10.43.115.162    Tue Feb 13 16:19:29 2018 - Tue Feb 13 16:22:00 2018  (00:02)    
tsliu2   pts/4        10.40.105.130    Tue Feb 13 16:17:21 2018 - Tue Feb 13 16:30:10 2018  (00:12)    
cwsmith  pts/13       10.40.91.247     Tue Mar 13 18:08:52 2018 - Tue Mar 13 18:46:52 2018  (00:38)    
asmith   pts/11       10.40.105.130    Tue Feb 13 14:07:43 2018 - Tue Feb 13 16:07:43 2018  (02:00)

It is always desirable to have a daily, weekly, or monthly usage reports by user or by remote host based on the above information.

Tasks for this assignment

In this assignment, your should preform the following activities:

  1. Complete a detail algorithm for producing daily, weekly, and monthly usage reports by user or by remote host based on the information stored in any given files generated from the 'last' command, either from a text file or real-time.
  2. Once you have complete the detail algorithm, you should then design the structure of your python script by identifying the appropriate python objects, functions and modules to be used for each task in your algorithm and the main control logic. Make sure to identify the followings:
    1. input data,
    2. computation tasks, and
    3. outputs.
  3. implement your computational solution using a single python script. You can use any built-in functions and functions from the python modules list in the "Allowed Python Modules" section below to implement your solution.
  4. Test and review your working python code to see whether you can improve the interface of each function to facilitate better code re-use (this process is called refactoring).

Allowed Python Modules

Instructions

Program Name and valid command line arguments

Name your Python3 script as ur_[student_id].py. Create a symbolic link to your script as ur.py (e.g. use the command ln -s ur_rchan.py ur.py to create the link) so that you can refer to your script as ur.py. Your script must accept one or more "file name" as its command line parameters and other optional parameters as shown below. Your python script should produce the following usage text when run with the --help option:

[rchan@centos7 a1]$ python3 ./ur.py -h
usage: ur_rchan.py [-h] [-l {user,host}] [-r RHOST] [-t {daily,weekly,monthly}]
             [-u USER] [-v]
             F [F ...]

Usage Report based on the last command

positional arguments:
  F                     list of files to be processed

optional arguments:
  -h, --help            show this help message and exit
  -l {user,host}, --list {user,host}
                        generate user name or remote host IP from the given
                        files
  -r RHOST, --rhost RHOST
                        usage report for the given remote host IP
  -t {daily,weekly,monthly}, --type {daily,weekly,monthly}
                        type of report: daily, weekly, and monthly
  -u USER, --user USER  usage report for the given user name
  -v, --verbose         tune on output verbosity

Copyright 2020 - Raymond Chan 

Replace the last line with your own full name


If there is only one file name provided at the command line, read the login/logout records from the contents of the given file. If the file name is "online", get the record on the system your script is being execute using the Linux command "last -iwF". The format of each line in the file should be the same as the output of 'last -Fiw'. Filter out incomplete login/logout record (hints: check for the number of fields in each record).

If there is more than one file name provided, merge all the files together with the first one at the top and the last one at the bottom. Read and process the file contents in that order in your program.

Header

All your Python codes for this assignment must be placed in a <b>single source file</b>. Please include the following declaration by <b>you as the script level docstring in your Python source code file (replace [Student_id] with your Seneca email user name, and "Student Name" with your own name):

OPS435 Assignment 2 - Winter 2020
Program: <b>ur_[seneca_id].py</b>
Author: "<font color='red'>Student Name</font>"
The python code in this file <b>ur_[seneca_id].py</b> is original work written by
"<font color='red'>Student Name</font>". No code in this file is copied from any other source 
including any person, textbook, or on-line resource except those provided
by the course instructor. I have not shared this python file with anyone
or anything except for submission for grading.  
I understand that the Academic Honesty Policy will be enforced and violators 
will be reported and appropriate action will be taken.

Sample outputs

The following are the reports generated by the usage report script (ur.py) with the "usage_data_file" mentioned in the overview section. You can download the file here to test your ur.py script.

User List

The following is the user list extracted from the usage_data_file created by the command:

[rchan@centos7 a2]$ ./ur.py -l user usage_data_file
User list for usage_data_file
=============================
asmith
cwsmith
rchan
tsliu2

Remote Host List

The following is the remote host list extracted from the usage_file_file created by the command:

[rchan@centos7 a2]$ ./ur.py -l host usage_data_file
Host list for usage_data_file
=============================
10.40.105.130
10.40.91.236
10.40.91.247
10.43.115.162

Daily Usage Report by User

The following is a Daily Usage Report created for user rchan by the following command:

[rchan@centos7 a2]$ ./ur.py -u rchan -t daily usage_data_file
Daily Usage Report for rchan
============================
Date          Usage in Seconds
2018 02 15        1980
2018 02 13        1580
Total             3560
[rchan@centos a2]$ ./ur.py -u cwsmith -t daily usage_data_file
Daily Usage Report for cwsmith
==============================
Date          Usage in Seconds
2018 03 13        2280
2018 02 15        7883
2018 02 14        3047
Total            13210

Daily Usage Report by Remote Host

The following is a Daily Usage Report created for the Remote Host 10.40.105.103 by the command:

[rchan@centos7 a2]$ ./ur.py -r 10.40.105.130 -t daily usage_data_file
Daily Usage Report for 10.40.105.130
====================================
Date          Usage in Seconds
2018 02 15        7883
2018 02 14        3047
2018 02 13        7969
Total            18899

Weekly Usage Report by User

The following is a Weekly Usage Report created for user rchan by the command:

[rchan@centos7 a2]$ ./ur.py -u rchan -t weekly usage_data_file
Weekly Usage Report for rchan
=============================
Week #        Usage in Seconds
2018 07           3560
Total             3560
[rchan@centos7 a2]$ ./ur.py -u cwsmith -t weekly usage_data_file
Weekly Usage Report for cwsmith
===============================
Week #        Usage in Seconds
2018 11           2280
2018 07          10930
Total            13210

Weekly Usage Report by Remote Host

The following is a Weekly Usage Report created for the remote host 10.40.105.130 by the command:

[rchan@centos7 a2]$ ./ur.py -r 10.40.105.130 -t weekly usage_data_file
Weekly Usage Report for 10.40.105.130
=====================================
Week #        Usage in Seconds
2018 07          18899
Total            18899

Suggested work-flow for this assignment

create a private repository on github

  • Use the same github account you used for your assignment 1 repository.
  • Create a private repository named "ops435-a2" for this assignment.
  • Populate your private repository with appropriate files. Please check out the sample repository here

Add collaborator to your ops435-a2 private repository

  • Add your professor's github account as one of the collaborators to your ops435-a2 private repository. This will allow your professor to pull the contents of your ops435-a2 repository and also to review and suggest changes and fixes to your algorithm and/or python script.
  • Make sure that your professor accepted your invitation from github.com.

Detail Algorithm Document

Follow the standard computation procedure: input - process - ouput when creating the algorithm document for this assignment.

input

  • get data (command line arguments/options) from the user using the functions provided by the argparse module
  • according to the arguments/options given at the command line, take appropriate processing action.

processing

  • based on the file(s) specified, read the contents of each file and use appropriate objects to store it
  • based on the command line arguments/options, process the data accordingly, which includes
    • data preprocessing (split a multi-day record into single day record)
    • record processing (preform required computation)

output

  • output the required report based on the processed data

identify and select appropriate python objects and functions

The following python functions (to be created, you may have more) are useful in handling the following sub-tasks:

  • reads login records from files and filters out unwanted records
  • convert login records into proper python object type so that it can be processed using as much built-in functions as possible
  • create function which generates daily usage reports by user and/or by remote host
  • create function which generates weekly usage reports by user and/or by remote host

To help you with this assignment, you can use the ur_template.py in the sample ops435-a2 repository as a starting point in designing your own Python Usage Report script. If you don't have enough time to create all the functions for the data processing steps, you should study the functions in the ur_funcs.py (provided by your teacher), pick and use the one that may help. If you use any of the functions from ur_funcs.py, there will be a cost of 10% to your overall grade. If you create all the functions yourself, you will get a bonus of 10%.

Python script coding and debugging

For each function, identify what type of objects should be passed to the function, and what type of objects should be returned to the caller. Once you have finished coding a function, you should start a Python3 interactive shell, import your functions and manually test each function and verify its correctness.

Final Test

Once you have all the individual function tested and that each is working properly, perform the final test with test data provided by your professor and verify that your script produces the correct results before submitting your python program on Blackboard. Upload all the files for this assignment 2 to your vm in myvmlab and perform the final test.

Sample login/logout records file and sample test run results

  • can be found from the sample repository github.com/rayfreeping/ops435-a2

Rubric

Task Maximum mark Actual mark
User Requirement Document 20
Program usage and Options 20
Generate user name list 10
Generate remote host IP list 10
Daily Usage Report by User 10
Daily Usage Report by Remote Host 10
Weekly Usage Report by User 10
Weekly Usage Report by Remote Host 10
Total 100

Submission

  • Stage 1: upload your algorithm document file to your ops435-a2 repository in github.com by March 20, 2020
  • Stage 2: upload your python script for this assignment to your ops435-a2 repository in github.com and to your vm in myvmlab by March 26, 2020
  • Stage 3: After fully tested and debugged your python script for this assignment, update your algorithm, your python script, and your est results to your ops435-a2 repository in github.com. Also submit the algorithm document, the python script and final test result to blackboard by April 3, 2020