The detail reference for the setting up all the configurations required for the Transaction Service is available on the click of the below link :
http://jbaileyinc.com/Downloads/Microsoft/AX/AXHeadquartersUserGuide.pdf
For setting up Retail Transaction Service:
http://technet.microsoft.com/en-us/library/hh580631.aspx
Besides this the Transaction Service can also be customized:
http://blogs.msdn.com/b/axsupport/archive/2012/01/09/ax-for-retail-customizing-the-retail-transaction-service.aspx
Also, for the Transaction Service troubleshooting, see below:
http://blogs.msdn.com/b/axsupport/archive/2012/09/19/unable-to-perform-pos-operations-requiring-transaction-service-error-occurs-regarding-the-clientchannelsink-obtainkeysforclient-operation.aspx