I am using the fx_mirrorWater() function from mtlfx.c and it works nicely. However, I was trying to see if I could combine this effect with the soft particles effect from the user wiki.

So far, I've been unsuccessful. I don't know anything about writing shaders, and I think by applying one material it will cancel the other one. So as a result I can have either a soft edge OR mirror water, but not both.

Now I know it's possible to have soft, reflective water in a game, there are plenty that do it. I tried combining the actual .fx files, but again, with no real knowledge of how shaders work, I didn't get very far.

Has anyone tried this, or know of a soft water shader that already exists?

You need to learn to write shaders in order to combine them and make some great looking graphics or use a complete graphics framework.

Gamestudio actually offers you a lot of comfort when writing shaders and pipelines. Also the shader tutorial as well as the manual are quite good in this concern. Just try it

Yes,
I guess I knew it would come to that. I found the shader workshop on the Gamestudio Downloads page ... is that a good place to start?

It is.
However, it could be a little more basic or give a better overview at first.
When you read that tutorial keep the following things in mind:
- A vertex shader gets executed for every vertex.
- A pixel shader is executed for every pixel drawn to the screen (independent of the model's texture resolution).
- You transfer data from the vertex shader to the pixel shader by writing the stuff into "texcoord" texture coordinate sets/ buffers. The data the pixel shader receives is interpolated data between the triangle's corresponding 3 vertices.
- All texture coordinates are in the [0,1] interval, independent of the actual resolution. (This means for example that when you use a [0,4] interval, f.i. by multiplying the "InTex" UV mapping data by 4, your texture will be tiled 4 tiles, assuming you use WRAP as an address mode (instead of CLAMP).)

Thanks for the advice!