This only happens when selling though, buying still works perfectly fine. Here I'll buy the whip pattern and it will remove 5 copper from me just like it says it should:
Now I'll sell the whip pattern back. Since I won't be getting any silver back for it, we can predict I won't be getting any copper either (even though I should get 5), and sure enough...
The server is 32bit, and I've gone and logged onto other servers and sold items just fine so I don't think it's a client issue. I have the price mod turned off and buycostmod and sellcostmod both set to 1.00 for testing purposes. This really has me stumped.