The Sitecore XML configuration has some fairly handy things in it, and ref is one of them. I spent a bit of time playing around with it some time ago; I think that I can provide an answer, but it's quite a long one so buckle in.
With 'how it works'... For fine detail, you can investigate the Factory.CreateObject(XmlNode, string[], bool, IFactoryHelper) and the Factory.CreateFromReference(XmlNode, string[], bool) methods, but in a nutshell Sitecore will attempt to construct the object in a few different ways. One of which is using a given ref or path attribute. Sitecore does not, at any point, actually fiddle with the underlying XML (e.g. by merging or copying it).
As you'll likely be aware, Sitecore assigns the properties of the object based on the child nodes present in the XML that identifies the type name. There are some exceptions to this case, but by and large this holds so long as the hint attribute is not used. To provide an example, the following XML (when requested through Factory) will create an object of type MyType and then proceed to set the value of the MyProperty property on that object to "foo":
<someNode type="MyType, MyAssembly">
<myProperty>foo</myProperty>
</someNode>
The population process described above occurs on two separate occasions: first with properties matching the child nodes present in the "referred" XML, and second with the properties matching child nodes present in the "original" XML. This is what allows for configuration such as with the Content Search indexes where index configuration can override a "default".
The interesting caveat here, is that because the XML is never merged, the "original" property values take precedence over the "referred" property values when used normally (i.e. set property X to value Y). This is why you often have to duplicate swathes of the default index configuration if you just want to add a value.
A further caveat to the above is: if you're using hint attributes to call methods then my understanding is that the method will be called for the "referred" XML and then the "original" XML. I haven't looked too deeply into this - but it makes sense given that it would be tricky to somehow "undo" the unknown effect of a method call. This is your best bet if you want to "merge" your values, and is probably why it only "sometimes" works if you just add in your changes rather than the full set of default values.
In terms of where it can be used, you can only use ref for XML that governs the instantiation of an object. Typically this means when you're creating an object for an object's property, or if you're making some top-level XML that you yourself intend to use the Factory class to instantiate.
To clarify the above: you - unfortunately - cannot use it if you're simply constructing an XML-only configuration node and want to reduce repetition, or to copy in commonly used values for properties (e.g. cache sizes for databases). You must be creating an object. I spent quite a bit of time attempting to do this and scratching my head when it didn't work, before digging a bit deeper.
A rather trivial point is that the ref will only be followed once per object. You cannot use a ref attribute to point to an element which is in turn another ref. Although you can obviously use ref on properties inside of those objects.
Hopefully that wasn't a whole lot of TL;DR and it answers your questions enough; if there's something I've missed or any mistakes please let me know.
