It IS possible to run a rotary motor on a ViX-xH drive.
In Easi_V software, select an ViX-xE as the product that you are working with (Utilities-----> Product). Using the guided servo setup option under the Utilities menu, generate the appropriate config file.
If you need to change the resolution of the feedback, go ahead and make the changes in the file that is generated.
The firmware looks at the Resolution and max speed and determines if the resulting input frequency is above 8 MHZ (post quad) it will not accept the MOTOR command.
If the resolution is too high, then lower the max speed parameter until (Resolution x Max Velocity) / 60