Litterbox training your rabbit
First and foremost, spay or neuter your rabbit. This will greatly increase the chances of your rabbit using a litterbox as well as make him calmer once the urge to mate is no longer a factor (as well as other benefits). The litterbox should be made of material that your rabbit cannot destroy and large enough for him to sit comfortably without touching either end. Hard plastic cat litterboxes are usually fine.
Your rabbit should first be trained to use a litterbox in his cage/hutch. A rabbit will usually eliminate in one area. Put a small amount of soiled hay into a litter box and put it where your rabbit is eliminating. Once he has started to use the litterbox you can gradually change from hay to organic litters made from alfalfa, oats or paper. Do not use softwood shaving (pine, cedar) as these may cause liver damage in rabbits.
Once your rabbit habitually uses the litterbox in his cage/hutch, you can start letting him out to hop around in a room. Always make sure the area is rabbit-proofed. Keep your rabbit in one area at first until he gets used to using the litterbox. Gradually increase the area he is allowed but also increase the litterboxes until they get used to the areas they are allowed in. If you have other pets make sure they are not a threat to your rabbit. You will have to slowly introduce them and have control over them so they don't attack. Learn more information about to safely introduce your rabbit to other pets in the house.
If your rabbit is marking his territory around his cage/hutch by leaving droppings but still uses the litterbox he may be trying to tell you that you may be invading his space. Try letting him come out on his own and go in on his own rather than picking him up out of his cage/hutch or putting him back in. Clean his cage/hutch daily but do it when he has chosen to be out of his cage/hutch. If you want him to get back in gently herd him or clap your hands softly to encourage him to go. If you have the rabbit on your lap and want him back in his cage/hutch, take him over to the cage but don't put him directly in it, let him hop inside on his own. Giving him the control of where he wants to be should alleviate the marking problems.