Difference between revisions of "Tutorial9: Regular Expressions"

From CDOT Wiki
Jump to: navigation, search
(LINUX PRACTICE QUESTIONS)
 
(414 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
===Main Objectives of this Practice Tutorial===
 
===Main Objectives of this Practice Tutorial===
  
:* Explain the purpose of '''Regular Expressions'''
+
:* Define the term '''Regular Expressions'''
  
 
:* Explain the difference between '''Regular Expressions''' and '''Filename Expansion'''
 
:* Explain the difference between '''Regular Expressions''' and '''Filename Expansion'''
Line 13: Line 13:
 
:* Understand and use command symbols for '''Extended''' Regular Expressions and their purpose
 
:* Understand and use command symbols for '''Extended''' Regular Expressions and their purpose
  
:* List various Linux commands that use regular expressions
+
:* List several Linux commands that can use regular expressions
 +
<br>
  
 
===Tutorial Reference Material===
 
===Tutorial Reference Material===
Line 37: Line 38:
  
  
|  style="padding-left:15px;"|Linux Commands  
+
|  style="padding-left:15px;"|Linux Commands
 +
* [https://ss64.com/bash/egrep.html egrep]
 
* [https://www.man7.org/linux/man-pages/man1/man.1.html man]
 
* [https://www.man7.org/linux/man-pages/man1/man.1.html man]
 
* [https://man7.org/linux/man-pages/man1/more.1.html more] / [https://www.man7.org/linux/man-pages/man1/less.1.html less]
 
* [https://man7.org/linux/man-pages/man1/more.1.html more] / [https://www.man7.org/linux/man-pages/man1/less.1.html less]
Line 50: Line 52:
 
= KEY CONCEPTS =
 
= KEY CONCEPTS =
  
<i>A '''regular expression''' … is a sequence of characters that define a search pattern.
+
===Regular Expressions===
  
Usually such patterns are used by string searching algorithms for "find" or "find and replace" operations on strings, or for input validation. </i><br>Reference:  https://en.wikipedia.org/wiki/Regular_expression
+
<i>A '''regular expression''' is a combination of two types of characters: '''literals''' and '''special characters'''.<br>Strings of text can be compared to this pattern to see if there is a match.</i>
  
===Regular Expressions vs. Filename Expansion===
+
This usually refers to text that is <u>contained</u> inside a '''file''' or text as a result<br>of issuing Linux commands using a '''Linux pipeline command'''.
 
+
<br><br>
<table align="right"><tr valign="top"><td>[[Image:re-1.png|thumb|right|350px|'''Filename expansion symbols''' allow the Linux shell to expand filenames as arguments (referred to as “globbing”).]]</td><td>[[Image:re-2.png|thumb|right|250px|'''Concept''' of matching a simple pattern of text contained within a text file.]]</td></table>In a previous lesson, you learned that '''filename expansion symbols''' allow the Linux shell to expand filenames as arguments (referred to as “globbing”) for file management commands. This is very useful for managing multiple files sharing similar characteristics such as the same file extension.
 
 
 
''Filename Expansion symbols'' are used to search, edit and manipulate text and are used with Linux file management commands such as '''ls''', '''rm''', '''mv''', '''cp''', '''cat''', '''less''' and '''more'''. This can represent text contained in files or text as a result of issuing Linux commands using a pipeline.
 
 
 
<i>A '''regular expression''' is a combination of two types of characters: '''literals''' and '''special characters'''.<br>In combination, these characters define a logical pattern. Strings of text can be compared to this pattern<br>to see if they fit the pattern defined by the expression.</i>
 
 
 
Reference: https://www.whoishostingthis.com/resources/regex/
 
  
 
===Literal (Simple) Regular Expressions===
 
===Literal (Simple) Regular Expressions===
  
[[Image:re-3.png|thumb|right|250px|The simplest regular expression is a series of letters and numbers, possibly including white space (tabs or space characters).]]The simplest regular expression is a series of letters and numbers, possibly including white space (tabs or space characters), that have no special meaning. Such a regular expression consists of ``literals''; that is, normal letters, which match only an identical letter in the data being searched.<br><br>For example:
+
[[Image:re-3.png|thumb|right|200px|A '''simple''' ('''literal''') regular expression is a series of letters and numbers (tabs or spaces).]]
"This is a regular expression"<br><br>
+
The simplest regular expression is a series of letters and numbers, (tabs or spaces).<br>A '''simple''' ('''literal''') regular expression consists of normal characters, which used to match patterns.<br><br>
<i>When an editor searches for a '''literal regular expression''', it can only score a ``hit'' if it finds exactly that sequence of characters in the data it is searching.</i> <br>Reference: http://osr507doc.xinuos.com/en/OSUserG/_Literal_chars_in_regexps.html
+
Although there are many Linux commands that use regular expressions, the '''grep''' command is a useful command to learn how to display matches of patterns of strings within text files.<br><br>
 +
For example:
 +
<span style="color:blue;font-weight:bold;font-family:courier;">grep Linux document.txt</span><br><br>
  
===Complex Regular Expressions===
+
=== Complex / Extended Regular Expressions ===
 
The problem with using literals or simple regular expressions is that only <u>simple</u> or <u>general</u> patterns are matched.
 
  
For example, the pattern “'''the'''” would be matched for larger word containing that pattern like '''there''', '''they''', '''either''', '''them''', '''their''', etc.<br>Another problem is that you may want to search for pattern at a specific location within the string of text (like at the beginning or end).
+
'''Complex Regular Expressions'''
 +
<br><br>
 +
The problem with just using '''simple''' ('''literal''') regular expressions is that only <u>simple</u> or <u>general</u> patterns are matched.  
  
There are other regular expression tools to provide more precise matches. These tools are '''complex''' and '''extended''' regular expressions.
+
''Complex Regular Expressions'' use symbols to help match text for more <u>precise</u> (complex) patterns.<br>The most common complex regular expression symbols are displayed below:
 
+
<br><br>
'''Complex Regular Expressions''' use symbols to help match text for more <u>precise</u> (complex) patterns.<br>The most common complex regular expression symbols are displayed below:
+
:'''Anchors: ''' <span style="color:blue;font-family:courier;font-weight:bold;">^</span> , <span style="color:blue;font-family:courier;font-weight:bold;">$</span><br>Match lines the begin (^) or end ($) with a pattern.<br>
 
+
:'''Single Character:''' &nbsp; <span style="color:blue;font-family:courier;font-weight:bold;">.</span><br>Represents a single character that can be any type of character.<br>
:*'''Anchors''' '''^''' , '''$'''
+
:'''Character Class:''' <span style="color:blue;font-family:courier;font-weight:bold;">[ ]</span> , <span style="color:blue;font-family:courier;font-weight:bold;">[^ ]</span><br>Represents a single character but with restrictions.<br>
:*'''Characters .'''
+
:'''Zero or More Occurrence:''' <span style="color:blue;font-family:courier;font-weight:bold;">*</span><br>Zero or more occurrences of previous character.<br><br>
:*'''Character Class [ ]''' , '''[^ ]'''
 
:*'''Zero or More Occurrence *'''
 
  
Examples of how to use these complex regular expressions with the grep command are displayed below:
+
:Examples of '''complex regular expressions''' are displayed below:
  
<table align="left"><tr valign="top"><td>[[Image:re-4.png|thumb|right|220px|Example of using '''anchors'''.]]</td><td>[[Image:re-5.png|thumb|right|200px|Example of matching by '''character(s)'''.]]</td><td>[[Image:re-6.png|thumb|right|220px|Example of using '''character class'''.]]</td><td>[[Image:re-7.png|thumb|right|200px|Example of matching '''zero or more occurrence of preceding character'''.]]</td></tr></table>
+
<table align="left"><tr valign="top"><td>[[Image:re-4.png|thumb|right|200px|Example of using '''anchors'''.]]</td><td>[[Image:re-5.png|thumb|right|175px|Example of matching by '''character(s)'''.]]</td><td>[[Image:re-6.png|thumb|right|220px|Example of using '''character class'''.]]</td><td>[[Image:re-7.png|thumb|right|200px|Example of matching '''zero or more occurrence of preceding character'''.]]</td></tr></table>
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
+
<br><br><br><br><br><br><br><br><br><br>
  
===Extended Regular Expressions===
 
 
   
 
   
'''Extended Regular Expressions''' consist of additional special characters the “extend” the capability of regular expressions.
+
'''Extended Regular Expressions'''
  
 +
''Extended Regular Expressions'' consist of additional special characters to “extend”<br>the capability of regular expressions. You must use the '''egrep''' or '''grep -E''' commands<br>in order to properly use extended regular expressions.
  
'''Repetition {min,max}'''<br>
 
Allows for more precise repetitions. Using braces, you can specify the '''minimum''' and/or '''maximum''' number of repetitions.
 
  
 +
:'''Repetition:''' <span style="color:blue;font-family:courier;font-weight:bold;">{min,max}</span><br>Allows for more precise repetitions. Using braces, you can specify<br>the '''minimum''' and/or '''maximum''' number of repetitions.
  
'''Grouping ( )'''<br>
+
:'''Groups:''' <span style="color:blue;font-family:courier;font-weight:bold;">( )</span><br>Allows you to search for repetition for a '''group of characters''', a '''word''', or a '''phase'''.<br>You enclose them within brackets <span style="font-family:courier;font-weight:bold;">( )</span> to specify a '''group'''.
Perhaps you want to search for repetition for a group of characters, a word, or a phase.<br>You can enclose them within brackets ( ) to specify a group.
 
  
 +
:'''or Condition:'''  <span style="color:blue;font-family:courier;font-weight:bold;">|</span><br>Can be used with '''groups''' to match a variety of character(s), words or phases.<br>The | symbol is used to separate the variety of character(s) within a ''group''.<br><br>
  
'''or Condition  |'''<br>
+
:Examples of how to use '''extended regular expressions''' with the '''egrep''' command are displayed below:<br><br>
Can be used with grouping to match a variety of character(s), words or phases. The | symbol is used to separate the variety of character(s).
 
  
Examples of how to use these complex regular expressions with the grep command are displayed below:<br><br>
+
<table align="left"><tr valign="top"><td>[[Image:re-8.png|thumb|right|280px|Example of using '''repetition'''.]]</td><td>[[Image:re-9.png|thumb|right|250px|Example of using '''groups'''.]]</td><td>[[Image:re-10.png|thumb|right|250px|Example of using '''or''' condition with '''groups'''.]]</td></tr></table>
 
+
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
<table align="left"><tr valign="top"><td>[[Image:re-8.png|thumb|right|280px|Example of using '''repetition'''.]]</td><td>[[Image:re-9.png|thumb|right|250px|Example of using '''grouping'''.]]</td><td>[[Image:re-10.png|thumb|right|250px|Example of using '''or condition with grouping'''.]]</td></tr></table>
 
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
 
  
 
=INVESTIGATION 1: SIMPLE &amp; COMPLEX REGULAR EXPRESSIONS=
 
=INVESTIGATION 1: SIMPLE &amp; COMPLEX REGULAR EXPRESSIONS=
  
 
<br>
 
<br>
In this section, you will learn how to use the '''grep''' command with '''simple and complex regular expressions''' to help search for patterns contained in text files.
+
In this investigation, you will learn how to use the '''grep''' command with '''simple and complex regular expressions'''<br>to help search for ''patterns'' contained in text files.
  
  
Line 120: Line 111:
  
 
# '''Login''' to your matrix account.<br><br>
 
# '''Login''' to your matrix account.<br><br>
# Issue a command to '''confirm''' you are located in your home directory.<br><br>The '''wget''' command is used to download files from the Internet to your shell.<br>This will be useful to download text files that we will be using for this tutorial.<br><br>
+
# Issue a Linux command to '''confirm''' you are located in your '''home''' directory.<br><br>The '''wget''' command is used to download files from the Internet to your shell.<br>This will be useful to download '''text files''' and '''data files''' that we will be using in this tutorial.<br><br>
# Issue the following linux command ('''copy and paste''' to save time):<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt</nowiki></span><br><br>
+
# Issue the following linux Linux command to '''download''' a text file to your '''home''' directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt</nowiki></span><br><br>
# Issue the '''ls''' command to confirm that the text file was downloaded.<br><br>
+
# View the contents of the '''textfile1.txt''' file using the '''more''' command see what data is contained in this file.<br><br>Although there are several Linux commands that use regular expressions,<br>we will be using the '''grep''' command for this investigation.<br><br>[[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the'''" (only lowercase). Notice the pattern matches larger words like "'''their'''" or "'''them'''".]]
# View the contents of the '''textfile1.txt''' file using the '''more''' command and quickly view the contents of this file. When finished, '''exit''' the more command.<br><br>Although there are several Linux commands that use regular expressions, we will only be using the '''grep''' command for this section.<br><br>
+
#Issue the following Linux command to match the pattern '''the''' within '''textfile1.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "the" textfile1.txt</span><br><br>Take a few moments to view the output and observe the matched patterns.<br><br>
#Issue the following linux pipeline command to match the pattern the within '''textfile1.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "the" textfile1.txt | more<br><br>
+
# Issue the grep Linux command with the <span style="font-weight:bold;font-family:courier;">-i</span> option to ignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -i "the" textfile1.txt</span><br><br>What do you notice is different when issuing this command?<br><br>You will notice that the pattern "'''the'''" is matched including larger words like "'''them'''" and "'''their'''".<br>You can issue the '''grep''' command with the <span style="font-weight:bold;font-family:courier;">-w</span> option to only match the pattern as a '''word'''.<br><br>
# Now, issue the grep linux pipeline command with the '''-i''' option to ignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -i "the" textfile1.txt | more</span><br><br>What do you notice is different with this pipeline command?<br><br>You will notice that the pattern "the" is matched including larger words that contain the pattern "the". You can use the -w option with the grep command in order to just match only words for a pattern.<br><br>
+
# Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the" textfile1.txt</span><br><br>You should now see only strings of text that match the word '''the''' (upper or lower case).<br><br>Matching literal or simple regular expressions can be useful, but are '''limited'''<br>in what pattens they can match. For example, you may want to<br>search for a pattern located at the '''beginning''' or '''end''' of the string.<br><br>There are other regular expression symbols that provide more '''precise''' search pattern matching.<br>These special characters are known as '''complex''' and '''extended''' regular expressions symbols.<br><br>For the remainder of this investigation, we will focus on '''complex regular expressions''' and then<br>focus on ''extended regular expressions'' in INVESTIGATION 2.<br><br><table align="right"><tr valign="top"><td>[[Image:regexps-2.png|thumb|right|280px|Anchoring regular expressions at the '''beginning''' of text.]]</td><td>[[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]</td></tr></table>
# Issue the following linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the" textfile1.txt | more</span><br><br>You should now see only strings of text that match the word '''"the"'''.<br><br>Matching literal or simple regular expressions can be useful, but are limited in what they can assist with pattern matching.<br>For Example, you may want to search for pattern at a specific location within the string of text (like at the beginning or end of the string).<br><br>There are other regular expression tools to provide more precise matches. These tools are '''complex''' and '''extended''' regular expressions. We will now look at complex regular expression symbols now, and we will discuss ''extended regular expressions''''''Italic text'''' in the next section of this tutorial.<br><br>
+
# Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the" textfile1.txt</span><br><br>The '''^''' symbol is referred to as an '''anchor'''.<br>In this case, it only matches<br>the word "'''the'''" (both upper or lowercase) at the <u>beginning</u> of the string.<br><br>
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the" textfile1.txt | more</span><br><br>The '''^''' symbol is an anchor. In this case, it only matches the <u>word</u> "the" (both upper or lowercase) at the beginning of strings.<br>The '''$''' symbol is used to anchor patterns at the end of strings.<br><br>
+
# Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the$" textfile1.txt</span><br><br>The '''$''' symbol is used to anchor patterns at the <u>end</u> of the string.<br><br>
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the$" textfile1.txt | more</span><br><br>What do you notice?<br><br>
+
# Issue the following Linux command to anchor the <u>word</u> "'''the'''"<br>'''simultaneously''' at the <u>beginning</u> and <u>end</u> of the string:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the$" textfile1.txt </span><br><br>What do you notice?<br><br>Anchoring patterns at both the <u>beginning</u> and <u>ending</u> of strings can greatly assist<br>for more '''precise''' search pattern matching.<br><br>We will now be demonstrate the '''effectiveness''' of <u>combining</u><br> '''anchors''' with <u>other</u> complex regular expressions symbols.<br><br><table align="right"><tr valign="top"><td>[[Image:regexps-4.png|thumb|right|280px|Anchoring regular expressions using '''period''' symbols at the '''beginning''' of text.]]</td><td>[[Image:regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]</td></tr></table>
# Issue the following Linux pipeline command to anchor the work "the" simultaneously at the beginning and the end of the string:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the$" textfile1.txt | more</span><br><br>What do you notice?<br><br>Anchoring patterns at both the <u>beginning</u> and <u>ending</u> of strings can greatly assist for more robust search patterns.<br>We will now be demonstrating '''simultaneous anchoring''' with other complex regular expressions symbols.<br><br>
+
# Issue the following Linux command to match strings that '''begin with 3 characters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^..." textfile1.txt</span><br><br>What do you notice? Can lines that contain '''less than 3 characters''' be displayed?<br><br>
# Issue the following command to match strings that begin with 3 characters:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^..." textfile1.txt | more</span><br><br>What do you notice?<br><br>
+
# Issue the following Linux command to match strings that '''begin <u>and</u> end with 3 characters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^...$" textfile1.txt</span><br><br>What do you notice compared to the previous command?<br><br>
# Issue the following command to match strings that begin and end with 3 characters:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^...$" textfile1.txt | more</span><br><br>What do you notice?<br><br>
+
# Issue the following Linux command to match strings that '''begin with 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]" textfile1.txt</span><br><br>What did you notice?<br><br>
# Issue the following command to match strings that begin with 3 digits:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]" textfile1.txt | more</span><br><br>
+
# Issue the following Linux command to match strings that '''end with 3 uppercase letters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "[A-Z][A-Z][A-Z]$" textfile1.txt</span><br><br><table align="right"><tr valign="top"><td>[[Image:regexps-6.png|thumb|right|220px|Anchoring '''3 digits''' at the '''beginning''' and '''ending''' of text.]]</td><td>[[Image:regexps-7.png|thumb|right|250px|Anchoring '''3 alpha-numeric characters''' at the '''beginning''' and '''ending''' of text.]]</td></tr></table>What type of strings match this pattern?<br><br>
# Issue the following command to match strings that end with 3 uppercase letters:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "[A-Z][A-Z][A-Z]$" textfile1.txt | more</span><br><br>
+
# Issue the following Linux command to match strings that '''consist of only 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]$" textfile1.txt</span><br><br>What did you notice?<br><br>
# Issue the following command to match strings that consist of only 3 digits:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]$" textfile1.txt | more</span><br><br>The '''*''' complex regular expression symbol is often confused with filename expansion. In other words, it does NOT represent zero or more of '''any character''', but zero or or occurrences of the character that comes before the * symbol.<br><br>
+
# Issue the following Linux command to match strings that '''consist of only 3 alphanumeric digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt</span><br><br>What did you notice?<br><br>The <span style="font-weight:bold;font-family:courier;">"*"</span> complex regular expression symbol is often confused with the "*"  '''filename expansion''' symbol.<br>In other words, it does NOT represent zero or more of '''any character''', but zero or more '''occurrences'''<br>of the character that comes '''before''' the <span style="font-weight:bold;font-family:courier;">"*"</span> symbol.<br><br>
# To demonstration, issue the following command to display zero or more occurrences of the letter x:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "x*" textfile1.txt | more</span><br><br>You will most likely notice most lines of the file is displayed.<br><br>
+
# To demonstrate, issue the following Linux command to display '''zero or more occurrences''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "x*" textfile1.txt</span><br><br>You will most likely notice most lines of the file is displayed.<br><br>
# Let's issue a command to display strings that contain more than one occurrence of the letter x:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "xx*" textfile1.txt | more</span><br><br>Why did this work? because the pattern indicates one occurrence of the letter x, followed by zero or MORE occurrences of the letter x.<br><br>If you combine the complex regular expression symbols .* it will act like zero or more occurrence of any character (like * did in filename expansion).<br><br>
+
# Let's issue a Linux command to display strings that contain '''more than one occurrence''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "xx*" textfile1.txt</span><br><br>Why did this work? because the pattern indicates one occurrence of the letter "x",<br>followed by '''zero or MORE occurrences''' of the <u>next</u> letter "x".<br><br>If you combine the complex regular expression symbols <span style="font-weight:bold;font-family:courier;">".*"</span> it will act like<br>zero or more occurrences of <u>any</u> character (i.e. like <span style="font-weight:bold;font-family:courier;">"*"</span> did in filename expansion).<br><br>
# Issue the following command to match strings begin and end with a number with nothing or anything inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9].*[0-9]$" textfile1.txt | more</span><br><br>Using '''simultaneous anchors''' combined with the .* symbol(s) can help you to refine your search patterns of strings.<br><br>
+
# Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9].*[0-9]$" textfile1.txt</span><br><br>Using '''simultaneous anchors''' combined with the <span style="font-weight:bold;font-family:courier;">".*"</span> symbol(s) can help you to refine your search patterns of strings.<br><br>
# Issue the following linux pipeline command to display strings that begin with a capital letter, ends with a number, and contains a capital X somewhere inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[A-Z].*X.*[0-9]$" textfile1.txt | more</span><br><br>Let's look at another series of examples involving '''filtering''' with numbers so only strings containing valid numbers are displayed.<br><br>
+
# Issue the following Linux command to display strings that begin with a capital letter,<br>end with a number, and contains a capital X somewhere inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[A-Z].*X.*[0-9]$" textfile1.txt</span><br><br>Let's look at another series of examples involving searching for strings that only contain '''valid numbers'''.<br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br>
# First, issue the following linux command to download another data file called '''numbers1.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat</nowiki></span><br><br>
+
# Issue the following Linux command to create the '''regexps''' directory: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/regexps</span><br><br>
# View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file. You should notice valid and invalid numbers contained in this file. When finished, exit the more command.<br><br>
+
# Change to the '''regexps''' directory and confirm that you have moved to this directory.<br><br>
# Issue the following linux command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | more</span><br><br>You may have noticed that the command does not entirely work. You may notice an empty line (which is NOT a whole number). This occurs since the * regular expression symbol represents ZERO or MORE occurrences of a number. You can use an additional numeric character class with the * regular expression symbol to search for one or more occurrences of a number.<br><br>
+
# First, issue the following Linux command to download another data file called '''numbers1.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat</nowiki></span><br><br>
# Issue the following linux command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9]*$" numbers1.dat | more</span><br><br>You should see that this works.<br><br>
+
# View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice '''valid''' and '''invalid''' numbers contained in this file. When finished, exit the more command.<br><br>
# Issue the following linux command to display whole positive or negative integers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | more</span><br><br>What did you notice?<br><br>
+
# Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | tee faulty.txt</span><br><br>You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line'''<br>(which is NOT a whole number). This occurs since the * regular expression symbol represents<br>ZERO or MORE occurrences of a number. You can use an additional numeric character class<br>with the * regular expression symbol to search for one or more occurrences of a number.<br><br>
# Issue the following linux command to display only whole numbers (with or without a positive or negative sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9]*$" numbers1.dat | more</span><br><br>
+
# Issue the following Linux pipeline command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt</span><br><br>You should see that this now works.<br><br>
 +
# Issue the following Linux pipeline command to display <u>only</u> '''signed''' integers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt</span><br><br>What did you notice? Positive and negative numbers display, not '''unsigned''' numbers.<br><br>[[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]
 +
# Issue the following Linux pipeline command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt</span><br><br>Did this command work?<br><br>
 +
# Issue the following command to check that you created those hard links: <br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/myscripts/week9-check-1</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.<br><br>
 +
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[0-9].*dat$"</span><br><br>What did this pipeline display?<br><br>
 +
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[a-z].*txt$"</span><br><br>What did this pipeline display?<br><br>
  
Proceed to Investigation 2.
+
: Although very useful, '''complex''' regular expressions do NOT <u>entirely</u> solve our problem of displaying<br> '''valid''' unsigned and signed numbers (not to mention displaying decimal numbers).<br><br>In the next investigation, you will learn how to use '''extended''' regular expressions that will completely solve this issue.<br>
 +
 
 +
: You can proceed to INVESTIGATION 2.
 +
<br>
  
 
=INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS =
 
=INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS =
  
In this section, you will learn how to use '''extended regular expressions''' to help refine your search when using regular expressions.
+
<br>
 +
In this investigation, you will learn how to use '''extended regular expressions''' with the '''egrep''' command<br>to further refine your search patterns.
  
  
 
'''Perform the Following Steps:'''
 
'''Perform the Following Steps:'''
  
# Issue the following linux pipeline command to download another data file called '''numbers2.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/numbers2.dat</nowiki></span><br><br>
+
# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
# View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice valid and more invalid numbers contained in this file. When finished, exit the more command.<br><br>
+
# Issue the following Linux command to download another data file called '''numbers2.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/numbers2.dat</nowiki></span><br><br>
# Issue the following linux pipeline command to display only whole numbers (with or without a positive or negative sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers2.dat | more</span><br><br>You should notice multiple + or - signs are appearing as well. This occurs since you are searching or one or MORE occurrences of a + or - sign.<br>Using '''extended regular expression''' symbols to specify minimum and maximum repetitions '''{min,max}''' can solve this problem.<br><br>
+
# View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice ''valid'' and ''invalid'' numbers contained in this file. When finished, exit the more command.<br><br>[[Image:eregexps-1.png|thumb|right|300px|'''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]]
# Issue the following linux pipeline command (using extended regular expression symbols) to display only whole numbers (with or without a positive or negative sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]{0.1}[0-9]{1,}$" numbers2.dat | more</span><br><br>'''NOTE:''' most likely, there were '''NO results'''. This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''. You would need to issue either '''grep -E''' (or more simply) issue the '''egrep''' command. The egrep command works with all regular expression symbols, and should be used in the future instead of the older grep command.<br><br>
+
# Issue the following Linux command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers2.dat</span><br><br>You should notice '''multiple''' '''+''' or '''-''' '''signs''' appear <u>prior</u> to some numbers.<br>This occurs since you are searching or one or MORE occurrences of a + or - sign.<br><br>Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}''' can solve that problem.<br><br>
# Reissue the above pipeline command using '''egrep''' instead of ''grep'':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | more</span><br><br>You should have noticed that the command worked correctly this time because you used the '''egrep''' command.<br><br>
+
# Issue the following Linux command (using extended regular expression symbols)<br>to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]{0.1}[0-9]{1,}$" numbers2.dat</span><br><br>'''NOTE: No output will be displayed! Why?'''<br><br>This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''.<br>You would need to issue either '''grep -E'''or just issue the '''egrep''' command. The egrep command works with<br>'''all''' regular expression symbols, and should be used in the future <u>instead</u> of the older grep command.<br><br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br>
# Issue the following linux pipeline command to display signed, unsigned, whole, and decimal numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | more</span><br><br>You can also use extended regular expression symbols for '''grouping'''. For example, you can search for repetitions of GROUPS of characters<br>(like a word) as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br>
+
# Issue the following Linux pipeline command using '''egrep''' instead of ''grep'':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt</span><br><br>[[Image:eregexps-2.png|thumb|right|300px|Using '''extended''' regular expression symbols (such as '''repetition''') to refine matches of ''signed'' and ''unsigned'' integers.]]You should have noticed that the command worked correctly this time because you used the '''egrep''' command.<br><br>'''NOTE:''' With extended regular expressions, the '''?''' symbol can be used to represent the '''{0,1}''' repetition symbols and the '''+''' symbol can be used to represent the '''{1,}''' repetition symbols<br><br>
 +
# Issue the following Linux pipeline command using the repetition shortcuts <span style="font-weight:bold;font-family:courier;">"+"</span> and <span style="font-weight:bold;font-family:courier;">"?"</span>:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt</span><br><br>You should have seen the '''same results''', but less typing was required.<br><br>
 +
# Issue the following Linux pipeline command to display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | tee better-number3.txt</span><br><br>Were all signed and unsigned intergers and decimal numbers displayed?<br><br>
 +
# Issue the follwoing command to check that you correctly issued<br>those ''Linux pipeline commands'': <br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/myscripts/week9-check-2</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use extended regular expression symbols for '''grouping'''.<br>For example, you can search for repetitions of GROUPS of characters (like a word)<br>as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br>
 
# Issue the following linux pipeline command to download another data file called '''words.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/words.dat</nowiki></span><br><br>
 
# Issue the following linux pipeline command to download another data file called '''words.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/words.dat</nowiki></span><br><br>
# View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice valid and more invalid numbers contained in this file. When finished, exit the more command.<br><br>
+
# View the contents of the '''words.dat''' file using the '''more''' command and quickly view the contents of this file.<br>Within this file, you should notice some lines that contain repetitions of words. When finished, exit the more command.<br><br>
# Issue the following linux pipeline command to display two or more occurrences of the word "the":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the){2,}" words.dat | more</span><br><br>'''NOTE:''' You should NOT see any output due to the fact that a <u>space</u> should be included at the end of the word "'''the'''". Usually words are separated by spaces; therefore, there were no matches since there were not occurrences of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern).<br><br>
+
# Issue the following linux pipeline command to display '''two or more occurrences''' of the word "the":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the){2,}" words.dat | tee word-search1.txt  more</span><br><br>'''NOTE: No output is displayed! Why?'''<br><br>This is due to the fact that a <u>space</u> should be included at the end of the word "'''the'''".<br>Usually words are separated by spaces; therefore, there were no matches since there were not occurrences<br>of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern).<br><br>
# Reissue the previous pipeline command including a space in brackets:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the ){2,}" words.dat | more</span><br><br>The "|" (or) symbol can be used within the grouping  symbols to allow matching of additional groups of characters.<br>Again, it is important to follow the character groupings with the space character<br><br>
+
# Reissue the previous pipeline command with the word the followed by a '''space''' within the brackets:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the ){2,}" words.dat | tee word-search2.txt</span><br><br>[[Image:eregexps-3.png|thumb|right|330px|Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of '''words''' (as opposed to ''characters'').]]The <span style="font-weight:bold;font-family:courier;">"|"</span> (or) symbol (same symbol as "pipe") can be used within the grouping  symbols to allow matching of additional groups of characters.<br>Again, it is important to follow the character groupings with the space character<br><br>
# Issue the following linux pipeline command to search for 2 or more occurrences of the word "'''the'''" <u>or</u> the word "'''and'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | more</span><br><br>
+
# Issue the following linux pipeline command to search for '''two or more occurrences''' of the word "'''the '''" <u>or</u> '''two or more occurrences''' of the word "'''and '''":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt</span><br><br>
 +
# Issue the following Linux command to check that you correctly issued<br>those ''Linux pipeline commands'' using the '''tee''' command to create those text files:<br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/myscripts/week9-check-3</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>Let's issue a Linux '''pipeline''' command using the '''egrep''' command as a '''filter'''<br>using <u>both</u> '''complex''' and '''extended''' regular expressions.<br><br>
 +
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | egrep "[a-z]{1,}.*[0-9]"</span><br><br>What did this Linux pipeline command display?<br><br>
 +
: The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions.<br>In the next investigation, you will apply regular expressions to a number of Linux commands<br>that you already learned in this course.
  
Proceed to Investigation 3
+
: You can proceed to INVESTIGATION 3<br>
 +
<br>
  
=INVESTIGATION 3: OTHER COMMANDS THAT USE REGULAR EXPRESSIONS =
+
=INVESTIGATION 3: OTHER COMMANDS USING REGULAR EXPRESSIONS =
 
+
<br>
In this section, you will see how regular expressions can be used with other Linux utilities other than '''grep''' or '''egrep'''.
+
In this investigation, you will see commands other than '''grep''' or '''egrep''' that can use regular expressions.
  
  
 
'''Perform the Following Steps:'''
 
'''Perform the Following Steps:'''
  
# Let's look at using regular expressions with the man command.<br>Issue the following linux command :<br><span style="color:blue;font-weight:bold;font-family:courier;">man ls</span><br><br>
+
# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
# We want to search for an option that can sort the file listing.<br>Type the following regular expression below and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/sort</span><br><br>'''FYI:''' The '''grep''' and '''egrep''' linux commands contain the regular expressions within quotes,<br>but most other Linux commands specify regular expressions using forward slashes<br>(e.g. '''/regular expression/''' or '''/regular expression''').<br><br>
+
# Let's look at using regular expressions with the '''man''' command.<br>Issue the following linux command :<br><span style="color:blue;font-weight:bold;font-family:courier;">man ls</span><br><br>[[Image:other-re-1.png|thumb|right|300px|Entering '''/sort''' in the '''man''' command can search for the string "'''sort'''".]]
# Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''"<br>(You can press '''SPACE''' and '''&lt;ALT&gt;&lt;b&gt;''' to move forward and backwards per page respectively).<br><br>
+
# We want to search for an option that can sort the file listing.<br>Type the following regular expression below and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/sort</span><br><br>'''FYI:''' The '''grep''' and '''egrep''' Linux commands contain the regular expressions within quotes, but '''most''' other Linux commands specify regular expressions using<br>'''forward slashes''' (e.g. <span style="font-weight:bold;font-family:courier;">/regular expression</span> &nbsp; or &nbsp; <span style="font-weight:bold;font-family:courier;">/regular expression/</span>).<br><br>
# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to exit the man pages for '''ls'''.<br><br>
+
# Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''"<br>(You can press '''SPACE''' or key combination '''alt-b''' to move forward and backwards one screen respectively).<br><br>
#Let's use regular expressions with the '''more''' command.<br>Issue the following linux command to view the contents of the '''textfile1.txt''' (downloaded in a previous section):<br><span style="color:blue;font-weight:bold;font-family:courier;">more textfile1.txt</span><br><br>
+
# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to '''exit''' the ''man'' pages for '''ls'''.<br><br>Let's use regular expressions with the '''less''' command.<br><br>
#We want to search for a pattern '''uli101''' within this text file.<br>Type the following regular expression and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>What did you notice?<br><br>
+
# Issue the following Linux command to download another data file called '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/large-file.txt</nowiki></span><br><br>[[Image:other-re-2.png|thumb|right|300px|Entering '''/uli101''' in the '''less''' command can display all matches of "'''uli101'''" throughout the text file.]]
#Search for the next occurrence of the pattern '''uli101''' by '''re-typing''' the following regular expression and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>you should now see the '''second occurrence''' of this pattern within the text file.<br><br>
+
# Issue the following Linux command to view the contents of the '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">less large-file.txt</span><br><br>
# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to exit the '''more''' command.<br><br>
+
#We want to search for a pattern '''uli101''' within this text file.<br>Type the following regular expression and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should see the pattern "uli101" throughout the text file.<br><br>
#Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility by using regular expressions.<br>Issue the following linux command to edit the '''textfile1.txt''' file:<br><span style="color:blue;font-weight:bold;font-family:courier;">vi textfile1.txt</span><br><br>Let's first perform a simple search within this text file.<br><br>
+
# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to exit the '''less''' command.<br><br>
# Type the following and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should move to the '''first occurrence''' of the pattern: '''uli101'''.<br><br>Let's search for the '''uli101''' pattern, but replace it in capitals (i.e '''ULI101''').<br><br>In vi, in order to perform and command, you need to go into last line mode, and then issue a command to apply to the entire text file, followed by a regular expression to search for and a regular expression to replace (i.e. '''/search/replace/''').<br><br>
+
# Try the same search techniques with the '''more''' command.<br><br>Does it work the same for the ''less'' command?<br><br>[[Image:other-re-3.png|thumb|right|300px|Entering '''/uli101''' in the '''vi''' command can search for the string "'''uli101'''".]]Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility<br>by using regular expressions.<br><br>
# Type the following and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101</span><br><br>You should have noticed that the first occurrence of uli101 has been changed to ULI101.<br><br>
+
# Issue the following Linux command to edit the '''large-file.txt''' file:<br><span style="color:blue;font-weight:bold;font-family:courier;">vi large-file.txt</span><br><br>Let's first perform a simple search within this text file.<br><br>
# Navigate throughout the text file to see if the other occurrences have been replaced.<br><br>You should notice they haven't for the other two occurrences. In order to replace for ALL occurrences, you need to add the letter g (meaning "'''global'''") at the end of the last forward slash (e.g /'''search/replace/g''').<br><br>
+
# Press the '''ESC''' key to make certain you are in '''COMMAND''' mode.<br><br>
# Making certain that you are command mode in vi, type the following and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101/g</span><br><br>
+
# Type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should notice the pattern "'''uli101'''" highlighted for ALL occurrences in this text file.<br><br>Let's '''search''' for the '''uli101''' pattern, and '''replace''' it in capitals (i.e '''ULI101''').<br><br>In vi, to issue a command, you need to enter '''LAST LINE''' MODE then issue a command.<br>Let's issue a command from '''LAST LINE''' MODE to search and replace '''uli101''' to '''ULI101'''.<br><br>[[Image:other-re-4.png|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressions to convert '''uli101''' to '''ULI101'''.]]
# Navigate throughout the text file to confirm that ALL occurrences of uli101 have been replaced with ULI101.<br><br>
+
# Making certain that you are '''COMMAND''' MODE in vi,<br>type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101/g</span><br><br>'''NOTE:''' The letter '''g''' after the replace regular expression represents "'''global'''" and will replace ALL occurrences of uli101 in the text document (as opposed to replacing the first occurrence for every line).<br><br>
# Save changes to your vi editing session and exit by typing the following and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:wx</span>
+
# Type the following (in uppercase letters) and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/ULI101</span><br><br>You should notice the pattern "'''ULI101'''" highlighted for ALL occurrences in this text file.<br><br>
# After you complete the Review Questions sections to get additional practice, then work on your<br>online assignment 3.
+
# Navigate throughout the text file to confirm that ALL occurrences of '''uli101''' have been <u>replaced</u> with '''ULI101'''.<br><br>
 +
# Save changes to your vi editing session and exit by typing the following and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:x</span><br><br>
 +
: After you complete the '''Review Questions''' section to get additional practice, then work on your<br>'''online assignment 3'''.
 
<br><br>
 
<br><br>
  
Line 197: Line 206:
  
 
The purpose of this section is to obtain '''extra practice''' to help with '''quizzes''', your '''midterm''', and your '''final exam'''.
 
The purpose of this section is to obtain '''extra practice''' to help with '''quizzes''', your '''midterm''', and your '''final exam'''.
 
+
<br><br>
'''REVIEW QUESTIONS: SIMPLE &amp; COMPLEX REGULAR EXPRESSIONS'''
+
==REVIEW QUESTIONS: SIMPLE &amp; COMPLEX REGULAR EXPRESSIONS==
  
 
Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
 
Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
Line 213: Line 222:
  
 
<pre>
 
<pre>
plym fury 77 73 2500
+
Plym    fury   77     73     2500
chevy nova 79 60 3000
+
chevy   nova   79     60     3000  
ford mustang 65 45 10000
+
ford   mustang 65     45     10003
volvo gl 78 102 9850
+
volvo   gl     78     102     9850  
ford ltd 83 15 10500
+
ford   ltd     83     15     10507 
chevy nova 80 50 3500
+
chevy   nova   80     50     3503   
fiat 600 65 115 450
+
fiat   600     65     115     450    
honda accord 81 30 6000
+
honda   accord 81     30     6000
ford thundbd 84 10 17000
+
ford   thundbd 84     10     17000
toyota tercel 82 180 750
+
toyota tercel 82     180     755
chevy impala 65 85 1550
+
chevy   impala 65     85     1553
ford bronco 83 25 9500
+
ford   bronco 83     25     9505
 
</pre>
 
</pre>
  
 
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
 
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
  
1. <span style="font-family:courier;font-weight:bold">grep plym ~murray.saul/uli101/cars</span>
+
1. <span style="font-family:courier;font-weight:bold">grep plym ~murray.saul/uli101/cars</span><br>
 
+
2. <span style="font-family:courier;font-weight:bold">grep -i fury ~murray.saul/uli101/cars</span><br>
 
+
3. <span style="font-family:courier;font-weight:bold">grep “^[m-z]” ~murray.saul/uli101/cars</span><br>
2. <span style="font-family:courier;font-weight:bold">grep -i fury ~murray.saul/uli101/cars</span>
 
 
 
 
 
 
 
3. <span style="font-family:courier;font-weight:bold">grep “^[m-z]” ~murray.saul/uli101/cars</span>
 
 
 
 
 
 
 
 
4. <span style="font-family:courier;font-weight:bold">grep -i “^[m-z]” ~murray.saul/uli101/cars</span>
 
4. <span style="font-family:courier;font-weight:bold">grep -i “^[m-z]” ~murray.saul/uli101/cars</span>
 
+
5. <span style="font-family:courier;font-weight:bold">grep “3$” ~murray.saul/uli101/cars</span><br>
 
+
6. <span style="font-family:courier;font-weight:bold">grep -i “c.*5$” ~murray.saul/uli101/cars</span><br>
 
 
5. <span style="font-family:courier;font-weight:bold">grep “3$” ~murray.saul/uli101/cars</span>
 
 
 
 
 
 
 
6. <span style="font-family:courier;font-weight:bold">grep -i “c.*5$” ~murray.saul/uli101/cars</span>
 
 
 
  
  
Line 255: Line 249:
 
'''Part B: Writing Linux Commands Using Regular Expressions'''
 
'''Part B: Writing Linux Commands Using Regular Expressions'''
  
 +
Write a single Linux command to perform the specified tasks for each of the following questions.
  
 
7. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the pattern:
 
7. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the pattern:
the
+
the<br>
 
 
 
 
 
 
 
8. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the word:
 
8. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the word:
the
+
the<br>
 +
9. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin with a number.<br>
 +
10. Write a Linux command to display all lines in the file called '''~/text.txt''' that end with a letter
 +
(either upper or lowercase).<br>
 +
11. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin and end with a number.<br>
 +
12. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 characters that can be anything.<br>
 +
13. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 numbers.<br>
 +
14. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains 1 or more “C” characters.<br>
  
 +
<br><br>
  
 
+
==REVIEW QUESTIONS: REGULAR EXPRESSIONS (INCLUDING EXTENDED REGULAR EXPRESSIONS)==
9. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin with a number.
 
 
 
 
 
 
 
 
 
10. Write a Linux command to display all lines in the file called '''~/text.txt''' that end with a letter
 
(either upper or lowercase).
 
 
 
 
 
 
 
 
 
11. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin and end with a number.
 
 
 
 
 
 
 
 
 
 
 
12. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 characters that can be anything.
 
 
 
 
 
 
 
 
 
 
 
13. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 numbers.
 
 
 
 
 
 
 
 
 
 
 
14. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains 1 or more “C” characters.
 
 
 
<br><br>
 
'''REVIEW QUESTIONS: REGULAR EXPRESSIONS (INCLUDING EXTENDED REGULAR EXPRESSIONS)'''
 
  
 
Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
 
Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
Line 323: Line 290:
 
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
 
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
  
1. <span style="font-family:courier;font-weight:bold">grep "^[-+]" ~murray.saul/uli101/numbers.txt</span>
+
1. <span style="font-family:courier;font-weight:bold">grep "^[-+]" ~murray.saul/uli101/numbers.txt</span><br>
 
+
2. <span style="font-family:courier;font-weight:bold">grep "^[-+]*.[0-9]" ~murray.saul/uli101/numbers.txt</span><br>
 
+
3. <span style="font-family:courier;font-weight:bold">grep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br> &nbsp; &nbsp;(Why?)<br>
2. <span style="font-family:courier;font-weight:bold">grep "^[-+]*.[0-9]" ~murray.saul/uli101/numbers.txt</span>
+
4. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br>
 
+
5. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span><br>
 
+
6. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span><br>
 
 
3. <span style="font-family:courier;font-weight:bold">grep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br> &nbsp; &nbsp;(Why?)
 
 
 
 
 
 
 
4. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span>
 
 
 
 
 
 
 
5. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span>
 
 
 
 
 
 
 
6. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span>
 
  
  
Line 356: Line 309:
 
9. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences of the word “the” (upper or lower case).
 
9. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences of the word “the” (upper or lower case).
  
10. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences<br> &nbsp; &nbsp; of the word “the” <u>or</u> the word “but” (upper or lower case).
+
10. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences<br> &nbsp; &nbsp; &nbsp; of the word “the” <u>or</u> the word “but” (upper or lower case).
  
11. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with a minimum of 2 occurrences<br> &nbsp; &nbsp; and a maximum of 4 occurrences of the word “the” or the word “but” (upper or lower case).
+
11. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with a minimum of 2 occurrences<br> &nbsp; &nbsp; &nbsp; and a maximum of 4 occurrences of the word “the” or the word “but” (upper or lower case).
  
  

Latest revision as of 15:59, 16 March 2021

USING REGULAR EXPRESSIONS


Main Objectives of this Practice Tutorial

  • Define the term Regular Expressions
  • Explain the difference between Regular Expressions and Filename Expansion
  • Explain the purpose of Literal (Simple) Regular Expressions
  • Understand and use common symbols for Complex Regular Expressions and their purpose
  • Understand and use command symbols for Extended Regular Expressions and their purpose
  • List several Linux commands that can use regular expressions


Tutorial Reference Material

Course Notes
Linux Command/Shortcut Reference
YouTube Videos
Course Notes:


Regular Expressions


Linux Commands Brauer Instructional Videos:

KEY CONCEPTS

Regular Expressions

regular expression is a combination of two types of characters: literals and special characters.
Strings of text can be compared to this pattern to see if there is a match.

This usually refers to text that is contained inside a file or text as a result
of issuing Linux commands using a Linux pipeline command.

Literal (Simple) Regular Expressions

A simple (literal) regular expression is a series of letters and numbers (tabs or spaces).

The simplest regular expression is a series of letters and numbers, (tabs or spaces).
A simple (literal) regular expression consists of normal characters, which used to match patterns.

Although there are many Linux commands that use regular expressions, the grep command is a useful command to learn how to display matches of patterns of strings within text files.

For example: grep Linux document.txt

Complex / Extended Regular Expressions

Complex Regular Expressions

The problem with just using simple (literal) regular expressions is that only simple or general patterns are matched.

Complex Regular Expressions use symbols to help match text for more precise (complex) patterns.
The most common complex regular expression symbols are displayed below:

Anchors: ^ , $
Match lines the begin (^) or end ($) with a pattern.
Single Character:   .
Represents a single character that can be any type of character.
Character Class: [ ] , [^ ]
Represents a single character but with restrictions.
Zero or More Occurrence: *
Zero or more occurrences of previous character.

Examples of complex regular expressions are displayed below:
Example of using anchors.
Example of matching by character(s).
Example of using character class.
Example of matching zero or more occurrence of preceding character.












Extended Regular Expressions

Extended Regular Expressions consist of additional special characters to “extend”
the capability of regular expressions. You must use the egrep or grep -E commands
in order to properly use extended regular expressions.


Repetition: {min,max}
Allows for more precise repetitions. Using braces, you can specify
the minimum and/or maximum number of repetitions.
Groups: ( )
Allows you to search for repetition for a group of characters, a word, or a phase.
You enclose them within brackets ( ) to specify a group.
or Condition: |
Can be used with groups to match a variety of character(s), words or phases.
The | symbol is used to separate the variety of character(s) within a group.

Examples of how to use extended regular expressions with the egrep command are displayed below:

Example of using repetition.
Example of using groups.
Example of using or condition with groups.





















INVESTIGATION 1: SIMPLE & COMPLEX REGULAR EXPRESSIONS


In this investigation, you will learn how to use the grep command with simple and complex regular expressions
to help search for patterns contained in text files.


Perform the Following Steps:

  1. Login to your matrix account.

  2. Issue a Linux command to confirm you are located in your home directory.

    The wget command is used to download files from the Internet to your shell.
    This will be useful to download text files and data files that we will be using in this tutorial.

  3. Issue the following linux Linux command to download a text file to your home directory:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt

  4. View the contents of the textfile1.txt file using the more command see what data is contained in this file.

    Although there are several Linux commands that use regular expressions,
    we will be using the grep command for this investigation.

    Output of grep command matching simple regular expression "the" (only lowercase). Notice the pattern matches larger words like "their" or "them".
  5. Issue the following Linux command to match the pattern the within textfile1.txt:
    grep "the" textfile1.txt

    Take a few moments to view the output and observe the matched patterns.

  6. Issue the grep Linux command with the -i option to ignore case sensitively:
    grep -i "the" textfile1.txt

    What do you notice is different when issuing this command?

    You will notice that the pattern "the" is matched including larger words like "them" and "their".
    You can issue the grep command with the -w option to only match the pattern as a word.

  7. Issue the following Linux command:
    grep -w -i "the" textfile1.txt

    You should now see only strings of text that match the word the (upper or lower case).

    Matching literal or simple regular expressions can be useful, but are limited
    in what pattens they can match. For example, you may want to
    search for a pattern located at the beginning or end of the string.

    There are other regular expression symbols that provide more precise search pattern matching.
    These special characters are known as complex and extended regular expressions symbols.

    For the remainder of this investigation, we will focus on complex regular expressions and then
    focus on extended regular expressions in INVESTIGATION 2.

    Anchoring regular expressions at the beginning of text.
    Anchoring regular expressions at the ending of text.
  8. Issue the following Linux command:
    grep -w -i "^the" textfile1.txt

    The ^ symbol is referred to as an anchor.
    In this case, it only matches
    the word "the" (both upper or lowercase) at the beginning of the string.

  9. Issue the following Linux command:
    grep -w -i "the$" textfile1.txt

    The $ symbol is used to anchor patterns at the end of the string.

  10. Issue the following Linux command to anchor the word "the"
    simultaneously at the beginning and end of the string:
    grep -w -i "^the$" textfile1.txt

    What do you notice?

    Anchoring patterns at both the beginning and ending of strings can greatly assist
    for more precise search pattern matching.

    We will now be demonstrate the effectiveness of combining
    anchors with other complex regular expressions symbols.

    Anchoring regular expressions using period symbols at the beginning of text.
    Anchoring regular expressions using period symbols simultaneously at the beginning and ending of text.
  11. Issue the following Linux command to match strings that begin with 3 characters:
    grep "^..." textfile1.txt

    What do you notice? Can lines that contain less than 3 characters be displayed?

  12. Issue the following Linux command to match strings that begin and end with 3 characters:
    grep "^...$" textfile1.txt

    What do you notice compared to the previous command?

  13. Issue the following Linux command to match strings that begin with 3 digits:
    grep "^[0-9][0-9][0-9]" textfile1.txt

    What did you notice?

  14. Issue the following Linux command to match strings that end with 3 uppercase letters:
    grep "[A-Z][A-Z][A-Z]$" textfile1.txt

    Anchoring 3 digits at the beginning and ending of text.
    Anchoring 3 alpha-numeric characters at the beginning and ending of text.
    What type of strings match this pattern?

  15. Issue the following Linux command to match strings that consist of only 3 digits:
    grep "^[0-9][0-9][0-9]$" textfile1.txt

    What did you notice?

  16. Issue the following Linux command to match strings that consist of only 3 alphanumeric digits:
    grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt

    What did you notice?

    The "*" complex regular expression symbol is often confused with the "*" filename expansion symbol.
    In other words, it does NOT represent zero or more of any character, but zero or more occurrences
    of the character that comes before the "*" symbol.

  17. To demonstrate, issue the following Linux command to display zero or more occurrences of the letter "x":
    grep "x*" textfile1.txt

    You will most likely notice most lines of the file is displayed.

  18. Let's issue a Linux command to display strings that contain more than one occurrence of the letter "x":
    grep "xx*" textfile1.txt

    Why did this work? because the pattern indicates one occurrence of the letter "x",
    followed by zero or MORE occurrences of the next letter "x".

    If you combine the complex regular expression symbols ".*" it will act like
    zero or more occurrences of any character (i.e. like "*" did in filename expansion).

  19. Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween:
    grep "^[0-9].*[0-9]$" textfile1.txt

    Using simultaneous anchors combined with the ".*" symbol(s) can help you to refine your search patterns of strings.

  20. Issue the following Linux command to display strings that begin with a capital letter,
    end with a number, and contains a capital X somewhere inbetween:
    grep "^[A-Z].*X.*[0-9]$" textfile1.txt

    Let's look at another series of examples involving searching for strings that only contain valid numbers.
    We will use pipeline commands to both display stdout to the screen and save to files
    for confirmation of running these pipeline commands when run a checking-script later in this investigation.

  21. Issue the following Linux command to create the regexps directory: mkdir ~/regexps

  22. Change to the regexps directory and confirm that you have moved to this directory.

  23. First, issue the following Linux command to download another data file called numbers1.dat:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat

  24. View the contents of the numbers.dat file using the more command and quickly view the contents of this file.
    You should notice valid and invalid numbers contained in this file. When finished, exit the more command.

  25. Issue the following linux pipeline command to display only whole numbers (i.e. no + or - sign):
    grep "^[0-9]*$" numbers1.dat | tee faulty.txt

    You may have noticed that the command does not entirely work. You may notice an empty line
    (which is NOT a whole number). This occurs since the * regular expression symbol represents
    ZERO or MORE occurrences of a number. You can use an additional numeric character class
    with the * regular expression symbol to search for one or more occurrences of a number.

  26. Issue the following Linux pipeline command to display only whole numbers:
    grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt

    You should see that this now works.

  27. Issue the following Linux pipeline command to display only signed integers:
    grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt

    What did you notice? Positive and negative numbers display, not unsigned numbers.

    Simultaneous anchoring of regular expressions using character class and zero or more occurrences to display signed and unsigned integers.
  28. Issue the following Linux pipeline command to display signed or unsigned integers:
    grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt

    Did this command work?

  29. Issue the following command to check that you created those hard links:
    bash /home/murray.saul/myscripts/week9-check-1

    If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.

    You can also use the grep command using regular expression as a filter in pipeline commands.

  30. Issue the following Linux pipeline command:
    ls | grep "[0-9].*dat$"

    What did this pipeline display?

  31. Issue the following Linux pipeline command:
    ls | grep "[a-z].*txt$"

    What did this pipeline display?

Although very useful, complex regular expressions do NOT entirely solve our problem of displaying
valid unsigned and signed numbers (not to mention displaying decimal numbers).

In the next investigation, you will learn how to use extended regular expressions that will completely solve this issue.
You can proceed to INVESTIGATION 2.


INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS


In this investigation, you will learn how to use extended regular expressions with the egrep command
to further refine your search patterns.


Perform the Following Steps:

  1. Make certain that you are located in your ~/regexps directory on your Matrix account.

  2. Issue the following Linux command to download another data file called numbers2.dat:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/numbers2.dat

  3. View the contents of the numbers2.dat file using the more command and quickly view the contents of this file.
    You should notice valid and invalid numbers contained in this file. When finished, exit the more command.

    Weakness of complex regular expressions that do not limit the number of positive or negative signs.
  4. Issue the following Linux command to display signed or unsigned integers:
    grep "^[+-]*[0-9][0-9]*$" numbers2.dat

    You should notice multiple + or - signs appear prior to some numbers.
    This occurs since you are searching or one or MORE occurrences of a + or - sign.

    Using extended regular expression symbols to specify minimum and maximum repetitions: {min,max} can solve that problem.

  5. Issue the following Linux command (using extended regular expression symbols)
    to display signed or unsigned integers:
    grep "^[+-]{0.1}[0-9]{1,}$" numbers2.dat

    NOTE: No output will be displayed! Why?

    This is due to the fact that the grep command was NOT issued correctly to use extended regular expression symbols.
    You would need to issue either grep -E, or just issue the egrep command. The egrep command works with
    all regular expression symbols, and should be used in the future instead of the older grep command.

    We will use pipeline commands to both display stdout to the screen and save to files
    for confirmation of running these pipeline commands when run a checking-script later in this investigation.

  6. Issue the following Linux pipeline command using egrep instead of grep:
    egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt

    Using extended regular expression symbols (such as repetition) to refine matches of signed and unsigned integers.
    You should have noticed that the command worked correctly this time because you used the egrep command.

    NOTE: With extended regular expressions, the ? symbol can be used to represent the {0,1} repetition symbols and the + symbol can be used to represent the {1,} repetition symbols

  7. Issue the following Linux pipeline command using the repetition shortcuts "+" and "?":
    egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt

    You should have seen the same results, but less typing was required.

  8. Issue the following Linux pipeline command to display signed, unsigned, whole, and decimal numbers:
    egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | tee better-number3.txt

    Were all signed and unsigned intergers and decimal numbers displayed?

  9. Issue the follwoing command to check that you correctly issued
    those Linux pipeline commands:
    bash /home/murray.saul/myscripts/week9-check-2

    If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.
    If you receive a congratulation message that there are no errors, then proceed with this tutorial.

    You can also use extended regular expression symbols for grouping.
    For example, you can search for repetitions of GROUPS of characters (like a word)
    as opposed to just a single character or a GROUP of numbers as opposed to a single digit.

  10. Issue the following linux pipeline command to download another data file called words.dat:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/words.dat

  11. View the contents of the words.dat file using the more command and quickly view the contents of this file.
    Within this file, you should notice some lines that contain repetitions of words. When finished, exit the more command.

  12. Issue the following linux pipeline command to display two or more occurrences of the word "the":
    egrep -i "(the){2,}" words.dat | tee word-search1.txt more

    NOTE: No output is displayed! Why?

    This is due to the fact that a space should be included at the end of the word "the".
    Usually words are separated by spaces; therefore, there were no matches since there were not occurrences
    of "thethe" as opposed to "the the" (i.e. no space after repetition of the pattern).

  13. Reissue the previous pipeline command with the word the followed by a space within the brackets:
    egrep -i "(the ){2,}" words.dat | tee word-search2.txt

    Using extended regular expression symbols (such as grouping) to refine matches of repetition of words (as opposed to characters).
    The "|" (or) symbol (same symbol as "pipe") can be used within the grouping symbols to allow matching of additional groups of characters.
    Again, it is important to follow the character groupings with the space character

  14. Issue the following linux pipeline command to search for two or more occurrences of the word "the " or two or more occurrences of the word "and ":
    egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt

  15. Issue the following Linux command to check that you correctly issued
    those Linux pipeline commands using the tee command to create those text files:
    bash /home/murray.saul/myscripts/week9-check-3

    If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.
    If you receive a congratulation message that there are no errors, then proceed with this tutorial.

    Let's issue a Linux pipeline command using the egrep command as a filter
    using both complex and extended regular expressions.

  16. Issue the following Linux pipeline command:
    ls | egrep "[a-z]{1,}.*[0-9]"

    What did this Linux pipeline command display?

The grep and egrep Linux commands are NOT the only Linux commands that use regular expressions.
In the next investigation, you will apply regular expressions to a number of Linux commands
that you already learned in this course.
You can proceed to INVESTIGATION 3


INVESTIGATION 3: OTHER COMMANDS USING REGULAR EXPRESSIONS


In this investigation, you will see commands other than grep or egrep that can use regular expressions.


Perform the Following Steps:

  1. Make certain that you are located in your ~/regexps directory on your Matrix account.

  2. Let's look at using regular expressions with the man command.
    Issue the following linux command :
    man ls

    Entering /sort in the man command can search for the string "sort".
  3. We want to search for an option that can sort the file listing.
    Type the following regular expression below and press ENTER:
    /sort

    FYI: The grep and egrep Linux commands contain the regular expressions within quotes, but most other Linux commands specify regular expressions using
    forward slashes (e.g. /regular expression   or   /regular expression/).

  4. Scroll throughout the man pages for the ls command to view matches for the pattern "sort"
    (You can press SPACE or key combination alt-b to move forward and backwards one screen respectively).

  5. Press the letter q to exit the man pages for ls.

    Let's use regular expressions with the less command.

  6. Issue the following Linux command to download another data file called large-file.txt:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/large-file.txt

    Entering /uli101 in the less command can display all matches of "uli101" throughout the text file.
  7. Issue the following Linux command to view the contents of the large-file.txt:
    less large-file.txt

  8. We want to search for a pattern uli101 within this text file.
    Type the following regular expression and press ENTER:
    /uli101

    You should see the pattern "uli101" throughout the text file.

  9. Press the letter q to exit the less command.

  10. Try the same search techniques with the more command.

    Does it work the same for the less command?

    Entering /uli101 in the vi command can search for the string "uli101".
    Let's learn how to perform a simple search and replace within the vi utility
    by using regular expressions.

  11. Issue the following Linux command to edit the large-file.txt file:
    vi large-file.txt

    Let's first perform a simple search within this text file.

  12. Press the ESC key to make certain you are in COMMAND mode.

  13. Type the following and press ENTER:
    /uli101

    You should notice the pattern "uli101" highlighted for ALL occurrences in this text file.

    Let's search for the uli101 pattern, and replace it in capitals (i.e ULI101).

    In vi, to issue a command, you need to enter LAST LINE MODE then issue a command.
    Let's issue a command from LAST LINE MODE to search and replace uli101 to ULI101.

    In last line MODE in the vi text editor, issuing a command using regular expressions to convert uli101 to ULI101.
  14. Making certain that you are COMMAND MODE in vi,
    type the following and press ENTER:
    :%s/uli101/ULI101/g

    NOTE: The letter g after the replace regular expression represents "global" and will replace ALL occurrences of uli101 in the text document (as opposed to replacing the first occurrence for every line).

  15. Type the following (in uppercase letters) and press ENTER:
    /ULI101

    You should notice the pattern "ULI101" highlighted for ALL occurrences in this text file.

  16. Navigate throughout the text file to confirm that ALL occurrences of uli101 have been replaced with ULI101.

  17. Save changes to your vi editing session and exit by typing the following and pressing ENTER:
    :x

After you complete the Review Questions section to get additional practice, then work on your
online assignment 3.



LINUX PRACTICE QUESTIONS

The purpose of this section is to obtain extra practice to help with quizzes, your midterm, and your final exam.

REVIEW QUESTIONS: SIMPLE & COMPLEX REGULAR EXPRESSIONS

Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to simulate a quiz:

https://ict.senecacollege.ca/~murray.saul/uli101/uli101_command_practice_9a.docx

Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).


Part A: Display Results from Linux Commands using Simple & Complex Regular Expressions

Note the contents from the following tab-delimited file called ~murray.saul/uli101/cars:

Plym    fury    77      73      2500
chevy   nova    79      60      3000    
ford    mustang 65      45      10003
volvo   gl      78      102     9850    
ford    ltd     83      15      10507   
chevy   nova    80      50      3503    
fiat    600     65      115     450     
honda   accord  81      30      6000
ford    thundbd 84      10      17000
toyota  tercel  82      180     755
chevy   impala  65      85      1553
ford    bronco  83      25      9505

Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.

1. grep plym ~murray.saul/uli101/cars
2. grep -i fury ~murray.saul/uli101/cars
3. grep “^[m-z]” ~murray.saul/uli101/cars
4. grep -i “^[m-z]” ~murray.saul/uli101/cars 5. grep “3$” ~murray.saul/uli101/cars
6. grep -i “c.*5$” ~murray.saul/uli101/cars


Part B: Writing Linux Commands Using Regular Expressions

Write a single Linux command to perform the specified tasks for each of the following questions.

7. Write a Linux command to display all lines in the file called ~/text.txt that contains the pattern: the
8. Write a Linux command to display all lines in the file called ~/text.txt that contains the word: the
9. Write a Linux command to display all lines in the file called ~/text.txt that begin with a number.
10. Write a Linux command to display all lines in the file called ~/text.txt that end with a letter (either upper or lowercase).
11. Write a Linux command to display all lines in the file called ~/text.txt that begin and end with a number.
12. Write a Linux command to display all lines in the file called ~/text.txt that contains exactly 3 characters that can be anything.
13. Write a Linux command to display all lines in the file called ~/text.txt that contains exactly 3 numbers.
14. Write a Linux command to display all lines in the file called ~/text.txt that contains 1 or more “C” characters.



REVIEW QUESTIONS: REGULAR EXPRESSIONS (INCLUDING EXTENDED REGULAR EXPRESSIONS)

Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to simulate a quiz:

https://ict.senecacollege.ca/~murray.saul/uli101/uli101_command_practice_9b.docx

Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).

Part A: Display Results from Linux Commands using Regular Expressions

Note the contents from the following tab-delimited file called ~murray.saul/uli101/numbers.txt:

+123
---34
+++++++++++17
-45
45p8
25.6
11

Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.

1. grep "^[-+]" ~murray.saul/uli101/numbers.txt
2. grep "^[-+]*.[0-9]" ~murray.saul/uli101/numbers.txt
3. grep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt
   (Why?)
4. egrep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt
5. egrep "^[+-]?[0-9]+$" ~murray.saul/uli101/numbers.txt
6. egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~murray.saul/uli101/numbers.txt


Part B: Writing Linux Commands Using Regular Expressions

Write a single Linux command to perform the specified tasks for each of the following questions.


7. Write a Linux command to display all lines in the file called ~/data.txt that begins with 1 or more occurrences of an UPPERCASE letter.

8. Write a Linux command to display all lines in the file called ~/data.txt that ends with 3 or more occurrences of the number 6

9. Write a Linux command to display all lines in the file called ~/data.txt that begins with 2 or more occurrences of the word “the” (upper or lower case).

10. Write a Linux command to display all lines in the file called ~/data.txt that begins with 2 or more occurrences
      of the word “the” or the word “but” (upper or lower case).

11. Write a Linux command to display all lines in the file called ~/data.txt that begins with a minimum of 2 occurrences
      and a maximum of 4 occurrences of the word “the” or the word “but” (upper or lower case).