A .22 LR rifle is dirt-cheap to buy ammo for. About 2 or three cents a round if you buy it in bulk, no recoil to speak of, not much blast. But of course wear hearing protection anyway.
The Model 60 is semi-automatic, with a tubular magazine. Hold 14 rounds (16 if you're creative), fires as fast as you can pull the trigger. Great fun at the range, especially against soda cans. I think my brother-in-law got one for $110 at Wal-Mart about three years ago.
http://www.marlinfirearms.com/Firearms/SelfLoading/60.aspxIf you like clips instead, you can get the Model 795. Also semi-automatic, but with a ten-round clip. I think this would be a better hunting gun because it would be either to load and unload getting in and out of cars. It might also be better in a self-defense emergency as well, for the same reason.
http://www.marlinfirearms.com/Firearms/SelfLoading/795.aspxThe Model 981T is a bolt-action tubular-magazine rifle. Same capacity as the Model 60. This ia good trainer if you're looking to practice working a bolt for a hunting rifle.
http://www.marlinfirearms.com/Firearms/BoltAction22/981T.aspxThe Model 925 is a bolt-action rifle with a 7-round clip.
http://www.marlinfirearms.com/Firearms/BoltAction22/925.aspxThe Golden 39A is a pretty sweet tubular-magazine lever-action rifle. My ex-wife's cousin had one, and I got to shoot it once. It was nice, very nice. It's a take-down gun, so you can break it down into two pieces for easy transport and cleaning.
http://www.marlinfirearms.com/Firearms/22Rifle/Golden39A.aspxYou can also step up to rifles that fire the .223 Remington. This is essentially the same cartridge that the military uses. The recoil is pretty light, unlike a shotgun, but it is a bit stiffer than a .22LR. It is a bit pricier, but you can get basic stuff for as little as 20 cents a round. Wolf Ammunition makes cheap .223 ammo.
Savage makes some bolt-action rifles like the Model 11FXP3, and they have have a reputation for being both moderately-priced and accurate.
http://www.savagearms.com/11fxp3.htmFor a quality .223 semiauto, the Ruger Mini-14 has a reputation for being extremely reliable. You can get (depending on the state you live in) high-capacity magazines as well, up to 40 rounds. The Mini-14 also has the ability to take the 5.56x45mm ammunition that the military uses, which means you can buy military-surplus stuff as well as the civilian brand ammo. You can also use a Mini-14 pretty effectively in the self-defense role, if need be.
http://www.ruger.com/Firearms/FAProdResults?function=famid&famid=20&variation=Mini-14®%20Ranch%20Rifle&bct=Yes&type=Rifle