“A parent does not have the right to aggress against his children, but also that the parent should not have a legal obligation to feed, clothe, or educate his children, since such obligations would entail positive acts coerced upon the parent and depriving the parent of his rights. The parent therefore may not murder or mutilate his child, and the law properly outlaws a parent from doing so. But the parent should have the legal right not to feed the child, i. e., to allow it to die. The law, therefore, may not properly compel the parent to feed a child or to keep it alive. (Again, whether or not a parent has a moral rather than a legally enforceable obligation to keep his child alive is a completely separate question.) This rule allows us to solve such vexing questions as: should a parent have the right to allow a deformed baby to die (e. g., by not feeding it)? The answer is of course yes, following a fortiori from the larger right to allow any baby, whether deformed or not, to die.”

Though, as we shall see below, in a libertarian society the existence of a free baby market will bring such "neglect" down to a minimum.
Children and rights, p. 100
The Ethics of Liberty (1982)