Expensive dot-sourcing

dot-sourcingToday a very short post about a concern that I would like to share. It all started at work, when our internal module went up to something like 12 seconds to load. And you my ask yourself: what is the big deal here? Well picture this:

Boss stands behind you and asks you: hey, do we have any JIRA tickets about this change you did last week? Can you find them for me?

Not expecting any issues, you type PowerShell command that should let you perform a quick JIRA search in CLI, proud of your own creation and simplicity of it. You type in the command name, a parameter that allows you to specify J<TAB>QL and… you wait…

… and wait…

“Bartek, why are you just staring at your monitor, I asked you a question…?”

… and wait…

“Excuse me?”

… and you open JIRA in the browser, because at least there it’s obvious that your doing something.

Forget lack of the WOW effect, you just look like a very disturbed person that doesn’t know how to use the tool he created. Your tab initiated loading of the module and froze your session for long, very long 12 seconds… Ouch.

That’s why (because of waiting, not because of any embarrassing situation like this one pictured) last week I decided to drill into this and to my big amazement majority (like 99.9%) of the time was spent on dot-sourcing all script files we have in our module. Don’t get me wrong, there were 83 of them (one per function). After trying few things I came to conclusion: it’s all about dot-sourcing a script. Namely: the fact that PowerShell will attempt to validate any file. You can see some details here, in Super User answer to a similar problem. It’s normally not a big hit as it is easy and quick in simple environments, but if your networks is a bit complex (proxy, that requires authentication and has only certain sites white-listed in our case) this process can be relatively slow. For one script, based on my tests (even if the script is almost empty) it takes around 140 milliseconds. That times 83…

Solution? There are few options (like pipe Get-Content to Invoke-Expression), but I ended up with dot-sourcing a script block created from content of the script. Here are both methods: one that was extremely slow for us that we were using before and the one that was very fast:

We went down to 300 milliseconds for loading whole module. But why do we use this design (each function in a separate script), you may ask? The answer is collaboration and git. But that’s a whole, different story. And nothing I could cover in this, short (yeah, right…) post! Smile

EDIT:

After receiving some feedback I noticed two things that could be improved in this approach. First one is encoding: I haven’t experienced any problems myself, but it’s probably a good idea to define encoding of the files that you read from disk.

Second problem I’ve already experienced first-hand. When you switch to dot-sourcing script blocks, you are no longer able to set break points on files where your functions are defined. To circumvent this we can use the fact that imported modules accept parameters:

I’ve already changed the logic in one of our modules at work to have the best of both worlds: quick loading by default and be able to debug module when speed is not so crucial.

EDIT # 2

This is one is based purely on the work done by Constatine Kokkinos (you can read about it on his blog). Long story short: dbatools was suffering from performance hit I described, so they’ve tried to use similar approach to improve loading time of their module. Unfortunately, it caused issues with Install-Module on WMF 5.0. We didn’t hit this bug at work because (by pure chance) we migrated to 5.1 just few weeks ago. It would hit us hard otherwise – we use internal PowerShell Gallery to distribute modules. I see in the commit history that it took few attempts to get it right, but final solution would look like this (source on GitHub):

I hope that nobody else experience similar problems once she/he updated their code using my suggestion. It worked for us, so I assumed it will work for everybody… If you did hit the wall like dbatools authors did – at least now thanks to them you know how to fix it…

Advertisements

8 thoughts on “Expensive dot-sourcing

  1. Performance wise this is spot on. However have you not now encountered issues with debugging? Whenever I try to use Set-PSBreakpoint or ISE breakpoints they simply do not work because im not setting breakpoints on the scriptblock you have created but the file it came from. I’d be interested to see if you encountered this and had a workaround

    • Thanks for input – I’ve updated the post/code to include that. I’ve acually faced the same problem myself and even though solution is not perfect – that’s the best I could come up with.

  2. Do not like. Why would you not just combine the files at build time into a single psm1? This method bypasses code signing just to get a little speed improvement — but you could get EVEN BETTER speed improvement by shipping a single file, without the side effect of breaking debugging and without exposing yourself to anyone who can manage to social-engineer someone into dropping a malicious script into your module folder…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s