Supporter languages: English (English )Spanish (Español )
Supporter Timezone: Europe/London (GMT+00:00)
You are right, custom CSS and JS added by Layouts is enqueued on *all* pages.
I have forwarded a feature request to the developers to be able to add CSS or JS on a per-Layout basis, so that custom JS would only be added to posts of a certain post type, for example. I can't say how long it would take to implement, I'm afraid, it depends very much on the other priorities of and demands on the Layouts team.
A minor performance improvement should be included in the next release which means the custom CSS file will not be enqueued if there is no custom CSS added.