They're only poisonous if you put the juice from the leaves or flowers in your mouth. Or if your fingers come into contact with the mucous membranes in the nose, eyes or mouth.
https://www.mountsinai.org/health-library/poison/foxglove-poisoning