Step 1: If you are on the machine that is running rlm.exe, run rlm.exe in admin mode

Step 2: Go to the browser and type http://localhost:5054. The RLM web server will look something like this:

Step 3: Then you can set it up as service by clicking in the "Manage Windows Service" on the left panel.

Step 4: Modify the following settings if required and click on Apply. 

Step 5: Upon successful installation of the service you can restart the system to get that into effect.

Refer to the RLM Documentation (RLM Documentation - Copyright (C) 2006, Reprise Software, Inc) Page 13 on "Running the rlm server as a service on Windows" for more details.