The gap in the wall around your pocket door can be a heat loss problem source, sending hot air straight up through the walls to the attic, and letting dust down. If you don't use the doors much anyway, why not consider sealing them with removable silicon and/or easily-removed trim? This will improve your air sealing in the home.