The short, safest and most efficient answer to this question is yes. According to the national code, with relation to electrical panels in closets: NEC 240.24D Overcurrent devices shall not be located in the vicinity of easily ignitable materials, such as a clothes closet. However, there are a few issues to consider a bout the language of this code and how it applies to your specific situation.
A quick Google search on the issue brings up a widespread debate among the Electrician Community about what can be properly defined as the “vicinity” or what constitutes a “closet." Some electricians might suggest that a panel located on the opposite wall of a clothes closet is still within the “vicinity." Some electricians argue that the definition of a “clothes closet” (as described specifically in the code) can easily be changed by removing the clothing rod and making the closet a space that doesn’t store easily ignitable material.
The bottom line is your family’s safety. An experienced and professional electrician wants to give you an honest assessment of your situation. If you have an electrical panel in a closet used to store clothes, you’re already taking an enormous risk. If a loose connection occurs in your existing electrical panel, sparks, those simple sparks can easily ignite the clothes hanging in your closet, and in a matter of seconds, set fire to your entire home.
Cutting corners to save money when it comes to electricity could easily be the worst decision you’ve ever made - electrical accidents are responsible for more home fires than almost any other cause. Upgraded and properly located electrical panels bring you peace of mind about your family’s safety, lower your electric bill costs, and increase your home’s energy efficiency.
The benefits of making proper electrical decisions will always outweigh the convenience of taking the easy way out.