It's come up in the past and you will be interested in this thread viewtopic.php?f=13&t=2137
It's not a major seecurity risk but it is a risk and you are correct to be hesitant. If you do decide to go ahead it could be worth creating a second Google account to use solely with GoogleVoice to mitigate the damage should the worst happen.
The 4 biggest risks listed in order of most easiest to hardest for your dialplan and hence any passwords it contains to be compromised are:
1. A sipsorcery administrator turns out to be a bad apple and harvests all sipsorcery accounts and sells off the details to the nearest black hat hacker. There is no way to complettely prevent that happening so all sipsorcery users are implicitly trusting sipsorcery administrators with their details. At this point there is only a single sipsorcery administrator, which is me, and if you've read the thread mentioned above then you shouldn't necessarily trust me because you don't know me.
2. An attacker uses social engineering to find our Google Voice username and runs a dictionary attack against your account to brute force your password. As much as it's a cliche if an attacker decides your details are worth something the easiest approach is to target the wekeest link in the chain which is invariably a human one.
3. A sipsorcery application server is compromised. If an attacker gained root access to a sipsorcery SIP server they could attach a debugger to the process that executes dialplans and capture any sensitive information. The sipsorcery SIP servers are running Windows 2003 and I have locked them down as tightly as possible, no file sharing or other dangerous services hanging around. Of course I could have missed something but I've been locking down Windows for a while so like to think I know what I'm doing.
4. The sipsorcery SQL Azure database gets compromised. At the moment all information is stored in the database in clear text so anyone with access to the database could get your Google Voice password. It is on my list [url](http://sipsorcery.codeplex.com/WorkItem ... temId=4147
[/url]) to encrypt sensitive information in the database but it's not done yet. Apart from that the probability of SQL Azure's security mechanisms being compromised is pretty low in my opinion. An advantage SQL Azure has is that unlike the case in a lot of applications the database infrastructure has been desinged and is being maintained by experts and the same company, Microsoft, that wrote the database software.
You also have the option of running the sipsorcery software locally on your own machine which alleviates 1,3 and 4 above. However it's not the most user friendly piece of software and will take you a bit of effort to get up and running.