In A- and late B-type stars, strong magnetic fields are always associated with Ap and Bp chemical peculiarities. However, it is not clear at what point in a star's evolution those peculiarities develop. Strong magnetic fields have been observed in pre-main sequence A and B stars (Herbig Ae and Be stars), and these objects have been proposed to be the progenitors of Ap and Bp stars. However, the photospheric chemical abundances of these magnetic Herbig stars have not been studied carefully, and furthermore the chemical abundances of 'normal' non-magnetic Herbig stars remain poorly characterized. To investigate this issue, we have studied the photospheric compositions of 23 Herbig stars, four of which have confirmed magnetic fields. Surprisingly, we found that half the non-magnetic stars in our sample show lambda Bootis chemical peculiarities to varying degrees. For the stars with detected magnetic fields, we find one chemically normal star, one star with lambda Boo peculiarities, one star displaying weak Ap/Bp peculiarities, and one somewhat more evolved star with somewhat stronger Ap/Bp peculiarities. These results suggests that Ap/Bp peculiarities are preceded by magnetic fields, and that these peculiarities develop over the pre-main sequence lives of A and B stars. The incidence of lambda Boo stars we find is much higher than that seen on the main sequence. We argue that a selective accretion model for the formation of lambda Boo peculiarities is a natural explanation for this remarkably large incidence.