Opinion of a relatively new brewer. I've made half a dozen or more batches of ginger beer now over the last year or so. I first went looking for recipes online. I found many, but none were exactly like I wanted (and somehow I hadn't heard of AHB, and the searches never brought it up). So I started from scratch. I took what looked good from different recipes and started my own.
My first recipe was much like your own. Ginger, Raw Sugar, Baker's yeast, juice and zest from lemons or limes, and that was it. It tasted alright, but it wasn't what I wanted. I played with the ratios of ingredients and it was a little better. Then I switched to US-05 yeast. That gave the single biggest improvement to the recipe to date, and it's not terribly expensive.
Then, I did a bit more searching around, talked to the guys at the local homebrew store, found this place, and my recipe has changed a little again. The latest brew was bottled a few days ago. I substituted some of the sugar for extra light malt extract, added a few hot chillies, and subbed a bit of fresh galangal for some of the ginger as an experiment.
Basically what I'm getting at is feel free to start with that base. See what it tastes like. It may not be horrible like some have suggested, but it will need some improvement. If you come up short, try playing around with the quantities of ingredients. If you feel you need to add more to get a greater depth of flavour, take inspiration from some of the other recipes floating around, because those ingredients have been proven to work well. My recipe isn't as complex as the others (yet), but it's constantly evolving, as I get it closer to my own tastes. But do yourself a favour and grab some US05 yeast and taste the difference next time. Do this one thing, if nothing else.
For what it's worth, my latest recipe: (15L experimental batch)
750g ginger (blended in food processor)
250g galangal (blended in food processor)
juice and zest from 2 large limes (my lime tree produces limes the size of lemons. You could use lemons instead)
4 medium sized hot chillies, finely chopped. Don't omit the seeds.
750g raw sugar
500g extra light malt extract
1/2 packet US05 yeast
OG 1030 FG 1006
Boil everything (except yeast!) in ~5L water, strain ingredients through muslin into fermenter, top with cold water to 15L. wrap up ginger pulp in muslin bag and place in fermenter. Pitch yeast, leave for 1-2 weeks until fermenting is complete. Decant liquid into another vessel to bulk prime, then bottle.
I haven't tasted the results yet, as it was only bottled a few days ago, but it tasted pretty good out of the fermenter. Warning: I like my GB dry; this recipe may not be to anyone's tastes but my own.