sed variable substitution

[andysimmons]

Hi Guys.

I'm having trouble getting sed to search/replace the value of a variable rather than the variable name itself. Here's the shell script...


let i=0

for j in $(cat $1); do
passArray[$i]=${j%:*}
let a=0

for n in $(cat $2); do
keyArray[$a]=${n%:*}
enc=$(echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 1)
dec=$(echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 2)
match=$(echo ${passArray[$i]} | sed -e 's/${enc}/${dec}/' | grep $dec)

if [ -n "$match" ]; then
echo $match >> $3
fi

let a=a+1
done

let i=i+1
done


This basically puts the passwords I was able to crack with another app into a text file along with some user information. Before I get flamed, it's all legit. We're doing this so we don't have to reset everyone's password when we create LDAP entries for them.

The stuff in bold is where I'm having trouble. I saw that bwkaz suggested double quotes in another thread for the substitution command, but that errors out saying it's an unknown option to s. Any suggestions would be greatly appreciated. Thanks.

[Stuka]

Is this Bash scripting? If so, shouldn't these lines enc=$(echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 1)
dec=$(echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 2) be more like enc=`echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 1`
dec=`echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 2` instead? I presume that you want enc and dec to contain the output from the echo/cut combo - the backticks do that, where the $() I'm not so sure of.
<edit> Fixed copy/paste spacing issue.

[andysimmons]

They've replaced backticks with $() to allow for nesting (i.e. $(command $(othercommand)) works, whereas `command `othercommand``doesn't). I appreciate the suggestion, but in this case they do the same thing. If I stick some echoes in there, I see that the variables are holding what I expect them to hold.

[sploo22]

Try changing ${enc} and ${dec} to $enc and $dec. It always works for me...

[andysimmons]

The $enc and $dec should work for pretty much anything, but since sed uses $ to search the end of a line, that format doesn't work with sed. Thanks for the suggestion though.

[Strogian]

match=$(echo ${passArray[$i]} | sed -e 's/${enc}/${dec}/' | grep $dec)

Okay, you say that bwkaz suggested you use double quotes, and that's right. That statement right there will not work (unless sed realizes that the $ shouldn't be interpreted as the end-of-line, in which case it will replace any literal '${enc}' with '${dec}'). If you use double-quotes instead, bash will replace ${enc} with the value of enc, and ${dec} with the value of dec.

The problem then must be what you have in those variables. What is some example data that might be in $enc of $dec? unless it all purely alphanumeric, you might run into trouble with sed interpreting part of it as a regular expression (or worse).

For instance:
enc='ar.to'
dec='rem/lep['
sed -e 's/ar.to/rem/lep[/'

See the problem? :)

There might be an option to sed to make it just do a simple find/replace, but I'm not aware of it. Maybe a different program...? It wouldn't take too much to write a simple find/replace program in C, actually, so that's always an option.. :)

[Stuka]

Just checking some online docs, and per them, the $ is NOT interpreted on the RHS of a regex in sed....

[Strogian]

Originally posted by Stuka
Just checking some online docs, and per them, the $ is NOT interpreted on the RHS of a regex in sed....

OK, what's the RHS? :) If it stands for "anything that isn't the last character of the regex," then that makes sense to me. :)

[andysimmons]

Ok, I got it...doubt it had anything to do with sed, just me being an idiot with the array assignments. I ended up using a temp file and a perl find and replace one-liner, and when it still screwed up I realized some colons were in the wrong spot.

Anyway, I'm not going to bother streamlining this. It's pretty inefficient, but just I'll post it just in case someone comes across this thread and needs the solution. Thanks for the info guys.

# /bin/bash

if [ -f $3 ]; then
rm $3
fi

let i=0

for j in $(cat $1); do

passArray[$i]=${j%*}
let a=0

for n in $(cat $2); do

keyArray[$a]=${n%*}
enc=$(echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 1)
dec=$(echo ${keyArray[$a]} | cut -d, --delimiter=":" -f 2)

echo ${passArray[$i]} > tmpfile
perl -pi -e "s/$enc/$dec/" tmpfile
match=$(grep $dec tmpfile)

if [ -n "$match" ]; then
echo $match >> $3
fi

let a=a+1
done

let i=i+1
done

if [ -f tmpfile ]; then
rm tmpfile
fi

[sploo22]

RHS = Right Hand Side

as in 's/LHS/RHS/' ;)

[dchidelf]

Originally posted by Stuka
Just checking some online docs, and per them, the $ is NOT interpreted on the RHS of a regex in sed....

I believe all that means is that $ won't represent an end of line on the replacement text side of the substitute.

The shell will interpret all variables in a " quoted string unless the $ is preceded by a \ or followed by something that is not a valid variable name. Korn and Bourne allow you to do things like ${variable}, I would assume bash does as well.

Originally posted by andysimmons
The $enc and $dec should work for pretty much anything, but since sed uses $ to search the end of a line, that format doesn't work with sed. Thanks for the suggestion though.

The $enc and $dec will disappear before sed even gets it's hands on the string, so you don't need to use ${enc} and ${dec}

$ VAR='\(.\)\(.*\)\1'
$ VAR2='\|\2\|'
$ echo ",hi,there" | sed "s/$VAR/$VAR2/"
|hi|there

What does --delimiter=":" do in cut?
My cut doesn't have that option.
How is it different than -d?

[andysimmons]

Originally posted by dchidelf
What does --delimiter=":" do in cut?
My cut doesn't have that option.
How is it different than -d?

First, thanks for the info, that's good to know. From what I can see, the --delimiter=":" is necessary (at least on this system) after the -d...
-d, --delimiter=DELIM use DELIM instead of TAB for field delimiter
Let me know if you know an easier way...I've learned most of this from reading books and occasionally posting, and the books don't always give shortcuts.

[dchidelf]

The cut that I use (a non GNU version), uses -d to tell it what delimiter to use.


echo "Hello:There:Goodbye:now" | cut -d: -f1,3
Hello:Goodbye


I'm guessing --delimiter=":" may just be a long option for -d:

[Stuka]

Originally posted by andysimmons
First, thanks for the info, that's good to know. From what I can see, the --delimiter=":" is necessary (at least on this system) after the -d...
That syntax means that -d and --delimiter are different forms of the same option. -d is the short form, and --delimiter is the long version.