I also, almost exclusively, use carbide drill bits. Just the other day, as crazy as it sounds, was freehand drilling some holes using #85 carbide bit. No, I didn't break it.
There are many sources of inexpensive carbide bits. No need to limit yourself Harbor Freight. I often buy mine on eBay. When I wand some specific sizes, I use
https://www.drillbitcity.com/Default.asp > Their resharpened bits are quite affordable (and like-new).
What sizes are you looking for? Few years ago I bought several large batches on eBay and I could let some of them go (in boxes of 50) for a really good price.