Okay, we're talking about an "arbor", right? It fits into the back of the drill chuck and the other end of it goes in the quill. It needs to be wiped really clean. I prefer to keep mine clean with acetone so it's clean and dead-dry before inserting it in the chuck. (The quill end actually has a threaded drawbar on mine, so it can't fall out on that end).
Well, I am definitely not a machinist and apparently haven't been using the proper names for the internal parts of the tools that I use to machine things. I called that "thingy" that fell our of my drill press (when I put to much side pressure on it) a "quill" because it was called a quill by somebody else on the Internet who said that is what could happen when you do what I did. And, frankly, it looks more like the quill of a feather than the part that is properly called a quill on a drill press. English is a weird language.
Anyway, I did dig out my instruction book for that drill press and now have a drawing with properly indicated names for the parts we are talking about, so that I can communicate more clearly, going forward. Interestingly, the instructions mark a drawing part with the name "quill", but the written material simply calls the spindle and quill "the spindle" and "the tapered hole in the spindle" for purposes of assembly. My arbor does
not have a threaded draw bar on the end that fits into the quill, so it
can fall out, and did, once. The chuck is also a press fit on the other end of the arbor, and is seated with "a tap" in my instructions, so I guess that could also come apart accidentally when using this press to do milling, but mine never has. Some of my other tools have chucks that screw-on, and of course, proper mills have threaded connections to avoid tapers working loose due to vibrations while forces are sideways instead of putting the joints in compression, as they are for drilling.
Although the instructions make it clear that the quill and arbor surfaces must be absolutely clear of particles and coatings to make a wobble-free connection, it only speaks to using a solvent "if the tapered hole is extremely dirty". I used mineral spirits, and hoped to leave a bit of residue to resist corrosion, because the arbor is supposed to remain removeable from the quill, and I was (am) concerned that moisture in the air could rust the mating surfaces over time and "freeze" them together if I used something like acetone to completely degrease them.
My experience with
gentle milling using this press has been good with the small burrs I originally posted. The time I dropped the arbor out of the quill was while (slowly) forming a 1/8" deep, 3"' diameter recess in an aluminum bar when making a tool to hold a boat propeller shaft still while torquing on a propeller puller on the other side of the hull. I just tried to take off too much material at one time. After I reassembled the arbor to the quill and took lighter cuts, I did succeed with no further problems.
That is not likely to be such a problem with the milling jobs needed to put a decoder into an N scale loco frame. So, my experience seems to indicate that you can use a reasonably solid drill press to do that kind of light milling, even if it does not have the threaded connections for the chuck to the arbor and the arbor to the spindle that a proper milling machine has.