Extract text between two strings repeatedly using sed or awk?

linux sed awk grep text-processing
Mike Furlender picture Mike Furlender · Nov 14, 2012 · Viewed 10.6k times · Source

I have a file called 'plainlinks' that looks like this:

13080. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94092-2012.gz
13081. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94094-2012.gz
13082. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94096-2012.gz
13083. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94097-2012.gz
13084. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94098-2012.gz
13085. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94644-2012.gz
13086. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94645-2012.gz
13087. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94995-2012.gz
13088. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-94996-2012.gz
13089. ftp://ftp3.ncdc.noaa.gov/pub/data/noaa/999999-96404-2012.gz

I need to produce output that looks like this:

999999-94092
999999-94094
999999-94096
999999-94097
999999-94098
999999-94644
999999-94645
999999-94995
999999-94996
999999-96404

Answer

Chris Seymour picture Chris Seymour · Nov 14, 2012

Using sed:

sed -E 's/.*\/(.*)-.*/\1/' plainlinks

Output:

999999-94092
999999-94094
999999-94096
999999-94097
999999-94098
999999-94644
999999-94645
999999-94995
999999-94996
999999-96404

To save the changes to the file use the -i option:

sed -Ei 's/.*\/(.*)-.*/\1/' plainlinks

Or to save to a new file then redirect:

sed -E 's/.*\/(.*)-.*/\1/' plainlinks > newfile.txt

Explanation:

s/    # subsitution
.*    # match anything
\/    # upto the last forward-slash (escaped to not confused a sed)
(.*)  # anything after the last forward-slash (captured in brackets)
-     # upto a hypen
.*    # anything else left on line
/     # end match; start replace 
\1    # the value captured in the first (only) set of brackets
/     # end

Related questions

What are the differences among grep, awk & sed?

Possible Duplicate: What are the differences between Perl, Python, AWK and sed? What is the difference between sed and awk? Maybe not a very specific question, but I am confused about the differences between grep, awk and sed in terms …

Read More
Command line: search and replace in all filenames matched by grep

I'm trying to search and replace a string in all files matched by grep: grep -n 'foo' * will give me output in the form: [filename]:[line number]:[text] For each file returned by grep, I'd like to modify the file …

Read More
grep for text with wild card in between

I would like to grep something like ==> *.sh <==. But it doesn't work, I can grep everything up to .sh <== but not get the wild card to work. What's the trick here?

Read More