Sed
sed (Wikipedia), or the stream editor, is a program which allows operations to be performed on a stream of input data. Among other uses, it is extremely useful for operations to change a file, which can be piped in.
sed is GNU software. Its homepage is here.
Contents |
Options
The -e option tells sed to add the following script to the list of commands to be executed. So we can run sed -e script or we can string multiples together like sed -e script1 -e script2 -e script3.
Usage Examples
To replace a pattern with a new pattern *globally*,
sed 's/wrong pattern/good pattern/g' wrong.file > good.file
in this case, wrong.file is the input and good.file is the output. Another way of doing the redirection would be:
cat wrong.file | sed 's/wrong pattern/good pattern/g' > good.file
To replace only the first occurrence on each line, remove the trailing "g", i.e.:
cat wrong.file | sed 's/wrong pattern/good pattern/' > good.file
Of course, it is always recommended to write to stdout and check output before writing to the file (i.e. remove the trailing output redirection ( > good.file)).
Dealing with printed variables/CSV
Occasionally, I have a script in a language such as Python that collects variables in a data structure such as a list, and ends with a statement to print the list. The result of this, in Python, is output like: ["value1", "value2", "value3", "value4"]. This isn't the easiest thing to look at. One option is to change the program to use a loop and print the values in a pretty way - one on each line. However, the advantage of outputting the whole list directly is that, if needed, it can be copied into another program, or read in from one line of a file, and acted on.
To convert the file to print nicely, we can use sed. For the above python-style syntax, we can simply split onto lines by replacing the commas with newline ("\n") characters:
cat test.txt | sed 's/,/\n/g'
Or do a version with output that's a bit nicer, but a bit of a kludge, in 3 parts:
1) Remove the brackets, replacing them with spaces (kludge):
cat text.txt | sed 's/]/ /;s/\[/ /g'
2) Replace the quotes with spaces. In this case, we double-quote the sed command so as not to confused it with the single quote:
cat text.txt | sed "s/'/ /g"
3) Split to multiple lines by changing commas into newlines:
cat text.txt | sed 's/,/\n/g'
4) Remove leading *and* trailing whitespace from all lines:
cat text.txt | sed 's/^[ \t]*//;s/[ \t]*$//'
We can add all of these together. However, as sed runs all commands inline, we cannot remove the whitespace in the same command. So, condensed, we have two commands :
Items 1-3: cat test.txt | sed -e 's/]/ /;s/\[/ /g' -e "s/'/ /g" > test2.txt
Item 4: cat test2.txt | sed -e 's/,/\n/g' -e 's/^[ \t]*//;s/[ \t]*$//'
This second command will send the output to stdout. You can also redirect it to a file.
