You may think it's just a minor touch-up that doesn't need any primer, but it does. Without it, the paint won't last. Try to apply some gray primer. The painter and decorator I work with told me it's the best in business when it comes to dark colors. If you don't want to invest in a primer, you could try to double coat them, but it'll happen the same thing. Usually, paint doesn't last without a primer. The surface needs to be prepared first. The extra-thick and sticky layer before the black color will definitely increase its endurance. Let me know if I have helped you. Perhaps you can show us some pictures of your little project!