All security measures are trade offs. Is the cost of the control worth more or less than the value of what is being protected, and how much more work does it force the attacker to do.
A lock on your luggage may stop a casual thief from opening your bag to grab something, but it won't stop someone stealing your bag. But the cost is low, so it is a still a useful security control.
So, just because a malicious agent could copy the referrer header, they may not have the time or inclination to do so. Think of a valid referrer header like a concert wristband. Can you forge them, or sneak around the guard checking wristbands? Sure you can, but it doesn't make the band useless.
It raises the cost for the attacker.
Likewise, your standard, non-modified, well-behaved application will tell the truth about the last page it came from, and then will correctly be denied API access. Maybe somebody trying to abuse your service doesn't know you are checking referrer headers on the server, and will give up.
So, in answer to OP's question "worthwhile" is a value judgement only you can make. How much effort does it take you to check referrer? Probably not much. How valuable is protecting use of your API key? Probably more. Is that amount of effort on your part worth the speedbump you'll place in the attacker's way?
That last one is your call.