I first installed basex using the brew script recommended under "Other Distributions" here:
basexgui worked fine, but when I ran basexhttp and went to localhost:8984 (and 127.0.0.1:8984) I got the following error:
"No function found that matches the request"
Rather than debug that, I uninstalled the brew installation and downloaded the
BaseX 8.5.3.zip to my Home directory (/Users/...).
Everything worked flawlessly out of the box, including basexhttp.
Mac users: Don't worry about no *.dmg file. In my experience, the zip file should be fine (assuming your system requirements are fine, like an up-to-date Java).