Working with JSON in Powershell - Collection of hashtables

Some time you interact with a rest API, which returns a block of json containing a collection of items that you want to enumerate over. If the collection of items is not in an array, but looks like the following, it is challenging to iterate over the collection.

{
    "firstItem" : [
        "one",
        "two"
    ],
    "next" : [
        "yes",
        "no",
        "maybe"
    ],
    "6739" : [
        "red",
        "blue",
        "green"
    ]
}

Of course we could say that the author of the rest api should change the returned result to make it easier to interact with, but that is not always within our control.

In PowerShell we can make this collection enumerable with a few extra lines of code. 

If you are working with a newer version of PowerShell (version 6 or newer) you can use the -AsHashtable parameter to ConvertFrom-JSON. 

If that is not the case, you can still make the json response enumerable as shown below. 

$json = ... response from the api
$data = $json | ConvertFrom-Json

$enumerableData = $data.psobject.properties | Select-Object Name, Value
foreach ($element in $enumerableData)
{
    Write-host "$($element.Name)"
    foreach ($element in $element.Value)
    {
	Write-Host "`t$element"
    }
}

The output from that is:

firstItem
        one
        two
next
        yes
        no
        maybe
6739
        red
        blue
        green

It is a nice little snippet that gets the job done.

PowerShell: Write-Output - What does it mean to pass objects along the pipeline

These days I spend a lot of time writing Terraform and PowerShell. There are many blogs about the difference between the various Write-xxx functions in PowerShell, Write-Host, Write-Verbose, Write-Error, Write-Output, etc. Many of these posts, such as Writing Output with PowerShell, as well as the actual documentation mention that Write-Output passes objects along the pipeline. This has an interesting side effect if you use Write-Output inside of a function in PowerShell. Let's play with some code to see what I mean. 

Let's start with a simple function.

This works as expected, we see the two lines in the output. Now let's make one change when we call the function

One line of the output is missing. This is because Write-Output passes the object along the pipeline. 

What if the function returns something, for example an array of strings? Let's take a look at the code and results in Visual Studio Code. I am hard coding the array here for illustrative purposes, I originally ran into this when I was querying an API and returning results in an array.    

Notice the returned $val, as shown on the left, has three elements in the array. The third is the parameter passed to Write-Output. The type of that is System.Management.Automation.PSObject. 

The challenge I ran into when I found this was that I found the size of the Array in the calling function was larger than the size of the array in the called function. The fact that Write-Output writes objects to the pipeline means we should only be using it when we intend to process the resultant object in a pipeline. If not, I would stay away from Write-Output and choose one of the other variants. If you have a preference of which variant is better and why, I'd love to hear your opinion. 

Azure DevOps Server (TFS) Operational Intelligence Activity Log

There is a little known feature of Azure DevOps Server, at least the on-prem version, that allows admins to look at the requests made to the server as well as the activity of the JobAgent.

This feature can be accessed by pointing your browser to 
<Base Server URL>/_oi/_diagnostics/activityLog
for example:
https://TFSServer/tfs/_oi/_diagnostics/activityLog


Once there you can switch between two views the Activity log and Job Monitoring by clicking on the tabs on the left.

The activity log shows a list of the requests being made to the server, including the identity of the requestor, the source IP address as well as the HTTP Response Code. I have found this helpful in tracking down when people are having issues.

One of the other nice things is that the activity log is exportable to CSV (see the green highlight above, which makes it very easy to work with in MS Excel and drill down to see just a single requestor.

The Job Monitoring tab shows a number of graphs about the Job Agent, the internal worker process. The graphs are all clickable and drill down to further information. 

I don't access this operational Intelligence information very often, but when I need it, it is information I can't find anywhere else. Hope you find it useful as well

Passing variables by Reference in PowerShell

I often have to write PowerShell scripts that perform some operation on a set of files. Sometimes I need to validate something about the files, for example that they are all well formed XML. 

The requirements are that 

1. All files are examined
2. Any files that have an issue are called out
3. The script returns a failure if one or more files fails the check.

To examine all of the files I use Get-ChildItem and then pipe that into a function to do the verification. 

Get-Childitem -path $directoryToCheck -filter '*.xml' -recurse | VerifyXMLFileFormat

and VerifyXMLFileFormat looks like

function VerifyXMLFileFormat
{
    foreach ($o in $input)
    {
        Write-host 'Reading:' $o.FullName

        try
        {
            [xml]$xml = Get-Content $o.FullName
        }
        catch
        {
            Write-Host "$($o.FullName) is not a valid XML file"
        }
    }
}

The challenge I have run into is that I have been unable to easily set a value in the called function and use it outside of the function because of how it is called as part of a pipeline. If in the catch block I use Write-Host-Error, or return $false, or throw, then not all of the files are processed. 

What I wanted was to pass a variable by reference so that VerifyXMLFileFormat can set the value and the calling function can reference the value. To do that I had to add a [REF] and access the .Value of the variable like this in the function

        catch
        {
            Write-Host "$($o.FullName) is not a valid XML file"
            $invalidFileFound.Value = $true
        }

Now I call the function like this

[bool] $invalidFileFound = false

Get-Childitem -path $directoryToCheck -filter '*.xml' -recurse | VerifyXMLFileFormat ([REF]$invalideFileFound)

Using that I can now meet all three requirements that I had

Can we avoid learned intuitiveness?

Learned intuitiveness, sounds like an oxymoron right? Learned intuitiveness is when you need to learn the general principles of how an interface works before it is intuitive (obvious) how to make it do what you want. In other words it is NOT automatically understood. Let's take a look at what Merriam Webster says about intuitiveness. There are two definitions that we should look at, the others are defined (in terms of intuitive)

1. directly apprehended
2. readily learned or understood

It is the later one that defines learned intuitiveness, it is easily understood, but perhaps not instantaneously. As an example of this, take an iPhone user and hand them and Android phone, or visa versa, and ask them to make a phone call. In my experience, unless one if familiar with both platforms, the user will fumble around trying to do what is natural for them on the platform they are used to because they learned that intuitiveness.

While this is wonderful for vendors because when users need to upgrade they will quite often stay with the platform they are familiar with due to their familiarity with it, they have learned the intuitiveness of that vendor's interface. In UX terms, upgrading to a new version of a platform should not break the users mental model. 

The question that I feel needs to be posed is do we need to rely on learned intuitiveness or can we build an interface that is automatically understood? I am not certain what the answer is and would love to hear your thoughts.

.NetCore - How to enable creation of a NuGet package during build

Before .Net Core if you wanted to generate a NuGet package you could use the csproj itself or a nuspec file, or actually both at the same time. Using a nuspec file, either by itself or in conjunction with the csproj allowed setting of some of the settings that could not be set easily, or at all, with just using the csproj.

With the introduction of the new style csproj files for .Net Core 2.0, the information that was previously stored in a nuspec file and in the assembly file can now be directly stored in the csproj file. Given a default csproj file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

</Project>

It is very easy to include the needed entries to generate a nupkg.

For example:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <Description>PUT THE DESCRIPTION HERE</Description>
    <Company>PUT THE COMPANY NAME HERE</Company>
    <Copyright>Copyright (c) PUT COPYRIGHT INFO HERE</Copyright>
    <Authors>PUT THE AUTHOR INFO HERE</Authors>
    <AssemblyName>PUT THE ASSEMBLY NAME HERE</AssemblyName>
    <PackageId>PUT THE PACKAGE ID HERE</PackageId>
    <PackageTags>PUT ANY APPLICABLE TAGS</PackageTags>
    <NeutralLanguage>en-US</NeutralLanguage>
    <Version>1.0.0</Version>
    <AssemblyVersion>1.0.0.0</AssemblyVersion>
    <FileVersion>1.0.0.0</FileVersion>

  </PropertyGroup>

</Project>

You will of course need to change the ALL CAP ITALICS entries above to the correct values for your project. You should have your automated build process  set the correct values for the three version entries.

There are more entries that can be added as per Additions to the csproj format for .NET Core such as adding the URL to the repository and the type of repository.

Unfortunately there is not an element that allows us to point to a readme file that will then show up in the NuGet GUI in Visual Studio. Because of this lack I hijack the PackageProjectURL element and put there the URL to the readme.md file that is required for all of my projects. If you do this remember to replace any ampersands in the URL with &amp; so that the URL link works correctly when displayed. Consumers of the NuGet package can then click on the link in the NuGet GUI to get more information. 

I find the .Net Core csproj much easier to deal with than the older project format. Teams should find it easier as it should lead to less merge conflicts simply because there is less in it. 

Should we always keep it DRY?

I am not talking about my basement, if I was the answer is a resounding YES! which I why I am having a waterproofing come up to the house to patch a hole in the foundation.

DRY in this case is an acronym for Don't Repeat Yourself.  This first place that I read about this principle was in The Pragmatic Programmer by Andrew Hunt and David Thomas, as far as I am aware they are the ones that made up the acronym DRY. In their words "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system." The goal is to avoid having multiple copies of code that diverge from each other, thereby causing bugs when a change is not applied to all copies.

As with all software development principles, it is important to ensure that we correctly apply this principle. As developers we need to be careful not to create duplicate code as this is, of course, undesirable. At the same time we need to make sure that we do not violate any architectural principles and we also need to be certain that we are looking at actual and not apparent duplication. At this point you may be wondering if I have lost my mind, isn't it fairly obvious if there are 2 methods in the same object of same hierarchy that have identical code. Yes that is obvious and today there are tools that will help us find and remove duplicated code. I am concerned that we remove the duplication in a way that doesn't decrease readability, that doesn't violate architectural design and that it is actual duplication.

I have seen too many developers, who apparently have had the DRY principle driven into their brains apply this relentlessly. In one case small sections of functions had between 2 and 4 lines of duplicated code across modules and architectural boundaries. The "fix" put in made the code significantly harder to read, (bad choices for method names didn't help) and added complexity to the code. Worse was not noting the architectural boundaries, the "fix" added a dependency between modules in the wrong direction across an architectural boundary. It is critical that we manage component and architectural dependencies. (This cannot be overstated and here I leave it as a single line in a blog...left an as exercise to the reader to understand why.)

Lastly, I mentioned apparent duplication. This happens much more often with data structures then method bodies. In apparent, but not real, duplication the two structures are logically separate and exist for separate reasons. Combining them will reduce maintainability because it will add dependencies. In most cases these structures will diverge due to feature requests. If they are forced together into the same structure then all interactions with that data structure will, over time, will not use all fields. One could argue force them together now and refactor later, based on architectural design that may or may not be appropriate. 

In short I just wanted to point out that relentless de-duplication is great as long as we keep in mind the larger architectural concerns of the software.