Expensive dot-sourcing

Today a very short post about a concern that I would like to share. It all started at work, when our internal module went up to something like 12 seconds to load. And you my ask yourself: what is the big deal here? Well picture this:

Boss stands behind you and asks you: hey, do we have any JIRA tickets about this change you did last week? Can you find them for me?

Not expecting any issues, you type PowerShell command that should let you perform a quick JIRA search in CLI, proud of your own creation and simplicity of it. You type in the command name, a parameter that allows you to specify J<TAB>QL and… you wait…

… and wait…

“Bartek, why are you just staring at your monitor, I asked you a question…?”

… and wait…

“Excuse me?”

… and you open JIRA in the browser, because at least there it’s obvious that your doing something.

Forget lack of the WOW effect, you just look like a very disturbed person that doesn’t know how to use the tool he created. Your tab initiated loading of the module and froze your session for long, very long 12 seconds… Ouch.

That’s why (because of waiting, not because of any embarrassing situation like this one pictured) last week I decided to drill into this and to my big amazement majority (like 99.9%) of the time was spent on dot-sourcing all script files we have in our module. Don’t get me wrong, there were 83 of them (one per function). After trying few things I came to conclusion: it’s all about dot-sourcing a script. Namely: the fact that PowerShell will attempt to validate any file. You can see some details here, in Super User answer to a similar problem. It’s normally not a big hit as it is easy and quick in simple environments, but if your networks is a bit complex (proxy, that requires authentication and has only certain sites white-listed in our case) this process can be relatively slow. For one script, based on my tests (even if the script is almost empty) it takes around 140 milliseconds. That times 83…

Solution? There are few options (like pipe Get-Content to Invoke-Expression), but I ended up with dot-sourcing a script block created from content of the script. Here are both methods: one that was extremely slow for us that we were using before and the one that was very fast:

	# Works fine in "normal" environment, can become slow when network is more complex...:
	foreach ($file in Get-ChildItem $PSScriptRoot\*.ps1) {
	. $file.FullName
	}
	# Should be fast everywhere, but it's bit more complex...
	foreach ($file in Get-ChildItem $PSScriptRoot\*.ps1) {
	. (
	[scriptblock]::Create(
	[io.file]::ReadAllText($file)
	)
	)
	}

view raw dotSourcing.ps1 hosted with ❤ by GitHub

We went down to 300 milliseconds for loading whole module. But why do we use this design (each function in a separate script), you may ask? The answer is collaboration and git. But that’s a whole, different story. And nothing I could cover in this, short (yeah, right…) post!

EDIT:

After receiving some feedback I noticed two things that could be improved in this approach. First one is encoding: I haven’t experienced any problems myself, but it’s probably a good idea to define encoding of the files that you read from disk.

Second problem I’ve already experienced first-hand. When you switch to dot-sourcing script blocks, you are no longer able to set break points on files where your functions are defined. To circumvent this we can use the fact that imported modules accept parameters:

	param (
	[bool]$DebugModule = $false
	)
	foreach ($file in Get-ChildItem $PSScriptRoot\*.ps1) {
	if ($DebugModule) {
	. $file.FullName
	} else {
	. (
	[scriptblock]::Create(
	[io.file]::ReadAllText(
	$file.FullName,
	[Text.Encoding]::UTF8
	)
	)
	)
	}
	}
	# To enable debugging - Import-Module path\to\Module -ArgumentList $true

view raw DotSource.psm1 hosted with ❤ by GitHub

I’ve already changed the logic in one of our modules at work to have the best of both worlds: quick loading by default and be able to debug module when speed is not so crucial.

EDIT # 2

This is one is based purely on the work done by Constatine Kokkinos (you can read about it on his blog). Long story short: dbatools was suffering from performance hit I described, so they’ve tried to use similar approach to improve loading time of their module. Unfortunately, it caused issues with Install-Module on WMF 5.0. We didn’t hit this bug at work because (by pure chance) we migrated to 5.1 just few weeks ago. It would hit us hard otherwise – we use internal PowerShell Gallery to distribute modules. I see in the commit history that it took few attempts to get it right, but final solution would look like this (source on GitHub):

	foreach ($file in Get-ChildItem $PSScriptRoot\*.ps1) {
	$ExecutionContext.InvokeCommand.InvokeScript(
	$false,
	(
	[scriptblock]::Create(
	[io.file]::ReadAllText(
	$file.FullName,
	[Text.Encoding]::UTF8
	)
	)
	),
	$null,
	$null
	)
	}

view raw InvokeScript.ps1 hosted with ❤ by GitHub

I hope that nobody else experience similar problems once she/he updated their code using my suggestion. It worked for us, so I assumed it will work for everybody… If you did hit the wall like dbatools authors did – at least now thanks to them you know how to fix it…

Advertisement