This is a question that comes up fairly often, so I decided to answer it here.
First of all, local sources:
The Target $1 section often has wooden instruments around Christmas, and sometimes at other points in the year. I wouldn't use these with any child who might put them in their mouth except under supervision, because they're labeled 3+, but they're an inexpensive, and often decent sounding way to build up an instrument collection. The wooden Castanets are especially nice.
Hobby Lobby has handbell sets, both small and large periodically, usually in the section right up by the cash register. These are the ones I use in class, and they are identical to those sold by music stores. Hobby Lobby often runs 40% off coupons online, so check there-with one, a set of handbells is only about $15.
Knowledge Tree, Village Toymaker, Toys R Us, Target Toy Section and Learning House all have a small instrument selection-but they're usually very expensive for what you get-and the sound quality isn't always great. Boomwhackers, the LP kids Percussion line, Woodstock percussion, Hohner, and Remo are sometimes available this way.
Your kitchen-empty containers, plastic storage containers, metal spoons...anything that's not heirloom, silver, or breakable can be used to make music!
Amro Music, Memphis Music and other Music stores. These sometimes carry the Remo or LP kids percussion, but they also carry small percussion instruments in general, and you can assume that these will generally be good instruments. My favorite brands for small percussion are Remo, LP, CP, Yamaha, and Nino. All have proven to be of high quality in the past.
The Kindermusik store sells many of the instruments used in class in small quantities to parents, and it's not a bad source for instruments. I do find that they're often a little expensive compared to other vendors, but Kindermusik safety tests their instrument to a much greater degree than almost anyone else out there-far more than what is required by law in the US, so especially for something your child will be playing with unsupervised, it's a good choice. Labeling on age tends to be conservative, so you can trust that if something is labeled for under 3s, it's going to stand up to being chewed, spit on, leaked on, thrown, pounded, and generally abused without breaking into sharp parts.
West Music, in Coralville Ia, is probably my favorite music store anywhere as far as comprehensive selection, and while they're often not the least expensive on everything, they are reasonably priced, have the materials in stock, so you rarely get back orders (I'm tempted to say never), and ship quickly. They are also one of the few vendors that sells the larger Orff instruments we sometimes use in class-which are expensive, but absolutely wonderful. They have a special "under 3 section", mostly the Hohner Kids, LP, and Remo line.
Empire music has lower prices than many other online stores, and has free shipping over $35. However, they do not sort based on age. "Early Childhood", in this case, means birth to age 6-7, so if you have a baby or toddler, I would not assume all of the instruments here are going to be safe for use for less supervised play.
Rhythm Band Inc has been around forever. If you played with those white resonator bars with red wood bases when you were in elementary school, or red wood drums with skin heads, or triangles with wood ball handles, you've probably played a rhythm band instrument. Most kindergarten classrooms have sets of their instruments. They are a good way to build up a collection quickly, but they're not as nice sounding as Remo, LP and the like. They ARE a great source for resonator bars and handbell resources, including nice color coded music. If they say something is "currently out of stock" or "currently back ordered" don't bother to order-you'll probably see it 6 months from now when you've completely forgotten about it. They do not ship quickly under the best of circumstances.
Suzuki has great prices on recorders, kids percussion, and Orff instruments. One thing I like about their Orff instruments is that they have holders for the extra bars and mallets right on the instrument, which is GREAT for home use. Please note that most of their "kids" instruments are rated 4+ unless they're reselling something made by another company.
Anyway, I hope this helps! Have fun making music!