On the hardness of approximating the permanent of structured matrices