Changes

Tutorial9: Regular Expressions

2,448 bytes added, 11:51, 7 March 2021

no edit summary

# '''Login''' to your matrix account.

# Issue a Linux command to '''confirm''' you are located in your '''home''' directory. The '''wget''' command is used to download files from the Internet to your shell. This will be useful to download '''text files''' and '''data files''' that we will be using for this tutorial.

# Issue the following linux Linux command to download a text file to your '''home ''' directory: wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt</nowiki>

# Issue the '''ls''' Linux command to confirm that the text file was downloaded.

# View the contents of the '''textfile1.txt''' file using the '''more''' command and quickly view the contents of this file. Although there are several Linux commands that use regular expressions, we will only be using the '''grep''' command for this investigation. [[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the'''" (only lowercase). Notice the pattern matches larger words like "'''their'''" or "'''them'''".]]

#Issue the following Linux command to match the pattern "'''the'''" within '''textfile1.txt''': grep "the" textfile1.txt Take a few moments to view the output and observe the matched pattern.

# Now, issue the grep Linux command with the -i option to ignore case sensitively: grep -i "the" textfile1.txt What do you notice is different when issuing this command? You will notice that the pattern "'''the'''" is matched including larger words like "'''them'''" and "'''their'''". You can issue the '''grep''' command with the -w option to only match the pattern as a '''word'''.

# Issue the following Linux command: grep -w -i "the" textfile1.txt You should now see only strings of text that match the word '''"the"''' (upper or lower case). Matching literal or simple regular expressions can be useful, but are '''limited''' in what they can assist with pattern matching. For example, you may want to search for a pattern located at the '''beginning''' or '''end''' of the string. There are other regular expression symbols that provide more '''precise''' pattern matches. These special characters are known as '''complex''' and '''extended''' regular expressions symbols. In this section, we will focus on complex ''regular expressions'' and then discuss ''extended regular expressions'' in INVESTIGATION 2. <table align="right"><tr valign="top"><td>[[Image:regexps-2.png|thumb|right|280px|Anchoring regular expressions at the '''beginning''' of text.]]</td><td>[[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]</td></tr></table># Issue the following Linux command: grep -w -i "^the" textfile1.txt The '''^''' symbol is referred to as an '''anchor'''. In this case, it only matches the word "'''the'''" (both upper or lowercase) at the beginning of the string.

# Issue the following Linux command: grep -w -i "the$" textfile1.txt The '''$''' symbol is used to anchor patterns at the end of the string.

# Issue the following Linux command to anchor the word "'''the'''" simultaneously at the beginning and end of the string: grep -w -i "^the$" textfile1.txt What do you notice? Anchoring patterns at both the beginning and ending of strings can greatly assist for more '''precise''' search patterns. We will now be demonstrate the power of anchoring combined with other complex regular expressions symbols. <table align="right"><tr valign="top"><td>[[Image:regexps-4.png|thumb|right|280px|Anchoring regular expressions using '''period''' symbols at the '''beginning''' of text.]]</td><td>[[Image:regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]</td></tr></table>

# Issue the following Linux command to match strings that begin with 3 characters: grep "^..." textfile1.txt What do you notice? Can lines that contain '''less than 3 characters''' be displayed?

# Issue the following Linux command to match strings that begin and end with 3 characters: grep "^...$" textfile1.txt What do you notice compared to the previous command?

# Issue the following Linux command to match strings that begin with 3 digits: grep "^[0-9][0-9][0-9]" textfile1.txt What did you notice?

# Issue the following Linux command to match strings that end with 3 uppercase letters: grep "[A-Z][A-Z][A-Z]$" textfile1.txt <table align="right"><tr valign="top"><td>[[Image:regexps-6.png|thumb|right|220px|Anchoring '''3 digits''' at the '''beginning''' and '''ending''' of text.]]</td><td>[[Image:regexps-7.png|thumb|right|250px|Anchoring '''3 alpha-numeric characters''' at the '''beginning''' and '''ending''' of text.]]</td></tr></table>Did any lines match this pattern?

# Issue the following Linux command to match strings that consist of only 3 digits: grep "^[0-9][0-9][0-9]$" textfile1.txt What did you notice?

# Issue the following Linux command to match strings that consist of only 3 alphanumeric digits: grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt What did you notice? The "*" complex regular expression symbol is often confused with the filename expansion symbol. In other words, it does NOT represent zero or more of '''any character''', but zero or more '''occurrences''' of the character that comes '''before''' the "*" symbol.

# Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign): grep "^[0-9]*$" numbers1.dat | tee faulty.txt You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line''' (which is NOT a whole number). This occurs since the * regular expression symbol represents ZERO or MORE occurrences of a number. You can use an additional numeric character class with the * regular expression symbol to search for one or more occurrences of a number.

# Issue the following Linux pipeline command to display only whole numbers: grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt You should see that this now works.

# Issue the following Linux pipeline command to display whole positive or negative integers: grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt What did you notice? Positive and negative numbers display, not not '''unsigned''' numbers. [[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]# Issue the following Linux pipeline command to display only whole numbers (with or without a positive or negative sign): grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt Did this command work? # Issue the following command to check that you created those hard links: bash /home/murray.saul/~~scripts~~myscripts/week9-check-1 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.~~ ~~If you receive a congratulation message that there are no errors, then proceed with this tutorial.

: Although very useful, '''complex''' regular expressions do NOT entirely solve our problem of displaying only '''valid''' unsigned and signed numbers. In the next investigation, you will learn how to use '''extended''' regular expressions that will completely solve this issue.

# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.

# Issue the following Linux command to download another data file called '''numbers2.dat''': wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/numbers2.dat</nowiki>

# View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file. You should notice valid and more invalid numbers contained in this file. When finished, exit the more command. [[Image:eregexps-1.png|thumb|right|300px|'''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]]

# Issue the following Linux command to display only whole numbers (with or without a positive or negative sign): grep "^[+-]*[0-9][0-9]*$" numbers2.dat You should notice '''multiple''' '''+''' or '''-''' '''signs''' appear prior to some numbers. This occurs since you are searching or one or MORE occurrences of a + or - sign. Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}''' can solve that problem.

# Issue the following Linux command (using extended regular expression symbols) to display only whole numbers (with or without a positive or negative sign): grep "^[+-]{0.1}[0-9]{1,}$" numbers2.dat '''NOTE:''' most likely, there were '''NO results'''. This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''. You would need to issue either '''grep -E''' (or more simply) issue the '''egrep''' command. The egrep command works with '''all''' regular expression symbols, and should be used in the future instead of the older grep command.

# Issue the following Linux pipeline command using '''egrep''' instead of ''grep'': egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt [[Image:eregexps-2.png|thumb|right|300px|Using '''extended''' regular expression symbols (such as '''repetition''') to refine matches of ''signed'' and ''unsigned'' integers.]]You should have noticed that the command worked correctly this time because you used the '''egrep''' command. '''NOTE:''' In extended regular expressions, the '''?''' symbol can be used to represent the '''{0,1}''' repetition symbols and the '''+''' symbol can be used to represent the '''{1,}''' repetition symbols

# Issue the following Linux pipeline command using the repetition shortcuts "+" and "?": egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt You should have seen the same results, but the extended regular expression required less typing.

# Issue the following Linux pipeline command to display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers: egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | tee better-number3.txt Were all signed and unsigned intergers and decimal numbers displayed?

# Issue the follwoing command to check that you correctly issued those ''Linux pipeline commands'': bash /home/murray.saul/~~scripts~~myscripts/week9-check-2 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial. You can also use extended regular expression symbols for '''grouping'''. For example, you can search for repetitions of GROUPS of characters (like a word) as opposed to just a single character or a GROUP of numbers as opposed to a single digit.

# Issue the following linux pipeline command to download another data file called '''words.dat''': wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/words.dat</nowiki>

# View the contents of the '''words.dat''' file using the '''more''' command and quickly view the contents of this file. You should notice valid and more invalid numbers contained in this file. When finished, exit the more command.

# Issue the following linux pipeline command to display two or more occurrences of the word "the": egrep -i "(the){2,}" words.dat | tee word-search1.txt more '''NOTE:''' You should '''NOT see any output''' This is due to the fact that a space should be included at the end of the word "'''the'''". Usually words are separated by spaces; therefore, there were no matches since there were not occurrences of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern).

# Reissue the previous pipeline command including a space in brackets: egrep -i "(the ){2,}" words.dat | tee word-search2.txt [[Image:eregexps-3.png|thumb|right|330px|Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of '''words''' (as opposed to ''characters'').]]The "|" (or) symbol (same symbol as "pipe") can be used within the grouping symbols to allow matching of additional groups of characters. Again, it is important to follow the character groupings with the space character

# Issue the following linux pipeline command to search for 2 or more occurrences of the word "'''the '''" or the word "'''and '''": egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt

# Issue the following Linux command to check that you correctly issued those ''Linux pipeline commands'' using the '''tee''' command to create those text files: bash /home/murray.saul/~~scripts~~myscripts/week9-check-3 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.

: The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions. In the next investigation, you will apply regular expressions to a number of Linux commands that you already learned in this course.

# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.

# Let's look at using regular expressions with the '''man''' command. Issue the following linux command : man ls [[Image:other-re-1.png|thumb|right|300px|Entering '''/sort''' in the '''man''' command can search for the string "'''sort'''".]]

# We want to search for an option that can sort the file listing. Type the following regular expression below and press '''ENTER''': /sort '''FYI:''' The '''grep''' and '''egrep''' Linux commands contain the regular expressions within quotes, but '''most''' other Linux commands specify regular expressions using forward slashes (e.g. /regular expression   or   /regular expression).

# Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''" (You can press '''SPACE''' or key combination '''alt-b''' to move forward and backwards one screen respectively).

# Press the letter q to '''exit''' the ''man'' pages for '''ls'''. Let's use regular expressions with the '''more''' command.

# Issue the following Linux command to download another data file called '''large-file.txt''': wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/large-file.txt</nowiki>

# View the contents of the '''large-file.txt''' file using the '''more''' command and quickly view the contents of this file. [[Image:other-re-2.png|thumb|right|300px|Entering '''/uli101''' in the '''more''' and '''less''' commands can search for the string "'''uli101'''".]]

# Issue the following Linux command to view the contents of the '''large-file.txt''': more large-file.txt

#We want to search for a pattern '''uli101''' within this text file. Type the following regular expression and press ENTER: /uli101 You should see the pattern "uli101" on the second line at the top.

#Search for the next occurrence of the pattern '''uli101''' by '''re-typing''' the following regular expression and pressing ENTER: /uli101 you should now see the '''second occurrence''' of this pattern near the top.

# Press the letter q to exit the '''more''' command.

# Try the same search techniques with the '''less''' command. Does it work the same for the ''less'' command as it did for the ''more'' command? [[Image:other-re-3.png|thumb|right|300px|Entering '''/uli101''' in the '''vi''' command can search for the string "'''uli101'''".]]

#Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility by using regular expressions. Issue the following Linux command to edit the '''large-file.txt''' file: vi large-file.txt Let's first perform a simple search within this text file.

# Press the '''ESC''' key to make certain you are in '''COMMAND''' mode.

# Type the following and press '''ENTER''': /uli101 You should move to the '''first occurrence''' of the pattern: '''uli101'''. Let's search for the '''uli101''' pattern, but replace it in capitals (i.e '''ULI101'''). In vi, ~~in order~~ to ~~perform and~~ issue a command, you need to ~~go into last line~~ enter '''LAST LINE''' mode~~, and~~ then issue a command. ~~to apply to the entire text file, followed by~~ Let's issue a ~~regular expression~~ command from LAST LINE mode to search ~~for~~ and ~~a regular expression~~ replace '''uli101''' to ~~replace (i.e.~~ '''~~/search/replace/~~ULI101'''). [[Image:other-re-4.png|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressions to convert '''uli101''' to '''ULI101'''.]]

# Making certain that you are command mode in vi, type the following and press ENTER: :%s/uli101/ULI101/g

# Navigate throughout the text file to confirm that ALL occurrences of uli101 have been replaced with ULI101.

Msaul

Administrators

13,420

edits

Changes

Tutorial9: Regular Expressions

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

get involved with CDOT

courses

course projects

links

Tools