Author: mw

Getting Started With Chocolatey 4 Business & Jenkins CI

I’ve had an argument with a colleague quite recently about the usage of open-source automation tools, especially Chocolatey in business environments. A keypoint of this argument was the integration of new/open tools in long-existing, mostly commercial software based workflows.

One of the main reasons to use Chocolatey in corporate environments is its ability to integrate seamlessly with already existent automation infrastructure. I’ve been able to integrate Chocolatey successfully with Jenkins CI, Finalbuilder, corporate internal windows services as well as simple ‘installer-batch-files’, a friend and coworker (flurl) is deploying a corporate internal management solution with Chocolatey.

The day I replaced the multi-100 line content of a legacy batch installer with a single line of

choco install {package} -y

was a really good day 🙂

In this blogpost I’m going to give you an idea of how you could get started with Chocolatey at your organization concerning two parts: a possible way to roll-out Chocolatey to your coworkers and a small collection of things to get started integrating Chocolatey with a CI-server (Jenkins, but probably applicable to anything else that does support PowerShell).

Deploying Chocolatey At Your Organization

You’ll want to make it as easy as possible, hence putting a .bat-file on a network share that’s accessible from everywhere in your org is probably a good idea.
If you’re just getting started with Choco I’d also suggest putting your first packages on such a share.
Let’s assume following share-layout:

  • \\your-server\Choco\install
  • \\your-server\Choco\packages

Go to https://chocolatey.org/packages and fetch the latest Chocolatey.nupkg file. Put that file together with this install.ps1 and this install.bat onto the Choco/install share.

You need to adapt the URLs and UNC-paths to the scripts inside the files to make them work in your environment.
Now you’re good to go, just double-click on install.bat from any machine and Chocolatey will be ready to use.
Notice that the community repository is removed during the installation progress and another (internal) repository is set up. For details on how this is done, check the contents of install.ps1.

Your First Packages

Doing a

choco list
now looks pretty boring, because there are no packages at your feed yet (the community repository has already been removed for corporate usage), so go ahead and download some .nupkg-Files from https://chocolatey.org/packages and put them into your  prepared network share.

Now you should be able to list the available packages with ‘choco list’, search for packages with ‘choco search’ and install packages with ‘choco install’.

Of course you can also create your own packages, just type

choco new {package name}
and see what happens.

So far, you’ve only been using OSS features, and you can use all this without limitation in a corporate environment, but there are some things you may need to consider: the packages you’ve pulled in from https://chocolatey.org probably rely on downloading external resources like .exe, .MSI or .zip files. You don’t want to do that in an organization, as you’ll want to have complete reliability in your solution, and downloading from the internet could have reliability issues (not to mention trust). There’s two options: manually download the required resources, changing the source code of the packages install.ps1 and repackage every single one of the required packages, or you could simple use package internalizer.

Package Internalizer

The best of the pack. If you’re using Chocolatey in a corporate environment, you’ll really dig this. Just type

choco download {packagename} --internalize
and let the magic happen. This will automatically download all needed assets and put them into the generated package, so you’ll achieve the maximum availability.

Just keep in mind that you’ll have to license all hosts that are consumers of packages you manage via internalization. (Chocolatey for Business starts at just $600 USD for up to 35 machines, and is $16/machine/year with volume discounts thereafter – see https://chocolatey.org/pricing)

Choco + Jenkins

Let’s automate the automation!

To be able to use the following jobs you’ll need to know the basics of Jenkins job configurations. You’ll need to set-up parameterized jobs with the PowerShell plugin, also Chocolatey should already be installed on the corresponding Jenkins nodes.

This job just updates all the Chocolatey packages on the server.

choco upgrade all -y

Yes, it’s as easy as that 🙂
This simple Jenkins-job allows you to internalize any package from the community-Feed with the click of a button:

# PowerShell script to internalize chocolatey packages from the community feed to an internal server using the business edition 'internalize' feature. This script is designed to be run from jenkins, P_* variables are defined in jenkins job!
# section CREDS
	$pkgs = $env:P_PKGLIST
	$uncshare = $env:P_UNC_SHARE
    $targetfolder = $env:P_DST_FOLDER
	$targetserver = $env:P_DST_SRV
	$apikey = $env:P_API_KEY

	$envtmp = $env:temp
	$tmpdir = "$envtmp\chocointernalize"
	$basefeed = "https://chocolatey.org/api/v2/"
# endsection

function InternalizePkg($pkg) {
	Push-Location $tmpdir
	choco download --internalize $pkg --resources-location="$uncshare\$pkg" --source="$basefeed" --no-progress
	$genpkg = ((Get-ChildItem *.nupkg -recurse).FullName | Select-String -Pattern $pkg)
    if ($targetfolder) {
        Write-Output "> copying package '$genpkg' to '$targetfolder'"
        if (-Not (Test-Path $targetfolder)) {
            New-Item $targetfolder -ItemType Directory -Force -Verbose            
        }
        Copy-Item $genpkg $targetfolder -Verbose -ErrorAction "Stop"
    } else {
        Write-Output "> pushing package '$genpkg' to '$targetserver'"
        choco push $genpkg --source="$targetserver" --api-key="$apikey" -Verbose
    }
    Write-Output "------------------------------------------------------------------------"
    Write-Output ""
	Pop-Location
}

if ((Test-Path $tmpdir)) {
	Remove-Item $tmpdir -Recurse -Force -Verbose
}
New-Item $tmpdir -ItemType Directory -Force -Verbose

$pkgs | ForEach-Object {
	InternalizePkg $_
}

Remove-Item $tmpdir -Recurse -Force -Verbose

 

This job reports what packages are outdated. (A newer version of that package is available at the community feed)


$mrecipient = $env:P_MAIL_REC
$msender = $env:P_MAIL_SEND
$msmtp = $env:P_SMTP_SERVER
$availPkgs = @()

#$chocoout = $(choco outdated)
$chocoout | ForEach-Object {
    $up = $_.Split("|")
    if ($up -And ($up[3] -ne "false")) {
			if ($up[1] -eq $up[2]) {
				$availPkgs += "$($up[0]): $($up[1]) -> $($up[2])"
			}
		} 
	
}

$res = $availPkgs | Format-Table | Out-String
if ($res) {
	Send-MailMessage -To $mrecipient -Subject "Chocolatey Packages Outdated!" -Body $res -Verbose -ErrorAction "Stop" -From $msender -SmtpServer $msmtp
}

Mind that this setup and these scripts are really minimalistic. When you’re working in an environment with a couple of users you may want additional feeds to create some logical package separation or access control. There are unimaginably many use-cases on how to interop with a command-line utility such as Chocolatey, but I just wanted to give you a quick dive-in how my first contact with Chocolatey looked like. (or how I wished it looked like to be honest ;-)).

Happy hacking!

Links:

PowerShell Logging Made Easy (thinking RAII)

Just a quick something I want to share :-)!
Make your life easier using a scope-based action approach when writing PowerShell.
The idea is simple: group tasks together in blocks that are passed into a function which logs start and end time of that block, as well as doing some logging and error reporting if something goes wrong during exection.

Let’s elaborate:

$start = Get-Date
Do-Something
Do-SomethingElse
$end = Get-Date
$duration = $end-$start
Write-FancyLog $duration "Done something"

This code doesn’t look nice. If you need to measure execution time of different blocks, do logging based on results, catch and evaluate errors, it would be much cleaner to do something like this:

FancyLog -Description "Do Something" {
  Do-Something
  Do-SomethingElse
}

Yes, that looks much better and is easier to grasp.

Without further detours, here’s a basic ‘logging monitor’ that nicely integrates into any PowerShell environment 🙂

function Monitor {
param(
  [Parameter(Mandatory = $True, ValueFromPipeline = $True)]
  [scriptblock]$scriptblk,
  [Parameter(Mandatory = $True)]
  [string]$Description,
  [Parameter(Mandatory = $False)]
  [scriptblock]$ResultReceiver,
  [Parameter(Mandatory = $False)]
  [switch]$CanFail
)
  $_start = Get-Date
  $_ex = $null
  $_res = 0
  try {
    & $scriptblk
    $res = $lastexitcode
  }
  catch {
    $_ex = $_
    $res = $lastexitcode
  }
  finally {
    $_resobj = @{
      Exception   = $_ex
      Result      = $_res
      Description = $Description
      Duration    = ($(Get-Date) - $_start)
		}
		if ($ResultReceiver) {
			$ResultReceiver.Invoke($_resobj) 
		}
    if (-Not $CanFail -And $_ex) {
      throw $_ex
    }
  }
}

Sample usage with dummy-logging:

function PseudoLog {
param(
  [Parameter(Mandatory = $True)]
  [PSCustomObject]$logobj
)
	Write-Host "> $($logobj.Description)"
	Write-Host $($logobj | ConvertTo-Json)
}

The following code continnues after reporting and errror:

Monitor -Description 'This block errs but continues!' -ResultReceiver $function:PseudoLog -CanFail {
	.\filesdoesnotexist
}

The following code breaks on the exception that is thrown, logs correctly and then rethrows the exception:

Monitor -Description 'This block will probably fail!' -ResultReceiver $function:PseudoLog {
	echo 'hello'
	throw 'foobar'
	echo 'cruel world'
}

Hope you enjoy this snippet, happy hacking!

Autumn In Austria – With My New Camera

Yup, I’ve finally put up the money to buy a brand new Nikon D810 – and I totally love it! The detail you get from that 36MP sensor just makes my trusty D3s look old and ‘dusty’.

I’ve started taking a couple of landscape shots with the 14-24 Nikkor lens I bought last year – a combination that is worth having.

Looking forward to as many as possibly ‘golden’, rainy and colorful autumn days in the next couple of months.

Komberg - Fields

 

Styrian Sunset

Back From Vacation

I’ve just returned from my 3-weeks vacation to the Azores – so there hasn’t been much going on here in the meantime 🙂
If you ever have the chance – the Azores are just an amazing place to see! We’ve traveled the islands Sao Miguel, Terceira, Pico as well as Faial, and all of them have at least those things in common:

  • beautiful landscape
  • great food
  • welcoming people

Here’s some pictures:

 

In the next couple of weeks I’ll be focusing on SoftwareAutomation and Invoke-Remote as much as possible, hopefully I’ll have time to put together post on this asap.

Using Chocolatey In Corporate Environments (Part #1)

It’s been about a year now that I’ve started pulling in Chocolatey as a tool for software deployment for the development teams at my workplace.
As of today, the setup of development machines, buildservers as well as specific terminal-hosts have been fully transformed from manually following many-pages of wiki install instructions to a single ‘meta-package’ that takes the hosts from 0 to 100 in ‘no-time’ (really, just like an hour compared to a full work-day previously!). And yes, all machines receive their very-own configuration, licensing for different software packages etc. etc. hyped? so am I!

Even though this is pretty obvious, it doesn’t hurt to recollect:
Having the install-procedure as well as the configuration in code (PowerShell in this case) makes it repeatable and less error prone! In addition to that, you’ll want to put your code under version control, so you’ve got the history covered!

Package Separation

For the sake of convenience I’ve decided to split up the packages that actually install software and the configuration of this software into separate packages.

vs2010
vs2010-settings-teamX
vs2010-settings-teamY
...

Internalize Everything!

In a business scenario, you probably don’t want to rely on public package source repositories (for multiple reasons) – so downloading the wanted packages, doing a virus-check and putting the artifacts on an internal-only server is basically what you want 99% of the time.
This can be a challenging task when done manually, trust me – I’ve been doing this and trying to automate it for about a month, but wait – this task already IS fully automated via the ‘internalize’ feature!
This feature is only available in the ‘business’ licensing model of Chocolatey, if you’re planning on using Chocolatey for work – go get a trial and let the magic work for you!

It is literally as easy as this:

choco download --internalize $pkg --source=https://repo...
choco push $pkg*.nupkg --source=https://myorg-choco... --api-key=$apikey

If you’re keen on going a step further, I’ve put up some sample ‘internalize’ script that I use to fetch packages via Jenkins CI. [internalize.ps1 on GitHub]

Completely Offline Chocolatey Installation

.\corporate_install.ps1 -dont -bug -me -with -external -resources

If you want to get into a lot of detail here, go check out the manual at chocolatey.org.

One thing that is currently missing (afaik) is a completely-offline Chocolatey installer.
You can always take the install.ps1 from chocolatey.org and modify it to your needs – but when you look at chocolateysetup.psm1 you’ll see that even a offline copy of chocolatey.nupkg could try to download a .Net Framework from a Microsoft mirror.

To overcome this rather annoying circumstance, I’ve decided to modify the base install.ps1 a little further to install a .Net Framework, if missing before installing Chocolatey – so no public internet connection is needed at any point.
I’ve put my efforts down into a script called corporate_install.ps1.
The script can be parametrized, so you should be able to fit it into your organization by just passing the right parameters!

You’ll also notice I’ve pulled Boxstarter, a great project adding reboot-resiliency (among many other cool features), into my base setup – this is because I use it all the time!

MIND! The first thing you’ll want to do when using chocolatey in a corporate environment is to remove the public community repository. (this is automatically done in my ‘corporate_install.ps1’ ;-))

choco source remove --name chocolatey

Artifact Server

Currently I’m using ProGet as artifact server, for solely two reasons:
1. It’s extremely easy to set-up and use
2. It’s got a feature called ‘package cache’

The package-cache feature pulls the wanted package from the HQ to the local package cache when first requested.
With this feature we set-up ‘location-based’ servers for our remote-offices. Now all employees are pulling packages from their ‘closest’ server – so no extra bandwith to HQ is needed.

Configuration Deployment & Software Licensing

This is probably the most ‘tricky’ bit from the currently existing setup and therefore will get an extra blogpost asap. Just note: we’ve created a simple ‘configuration service’ containing information about ‘software’ and ‘licensees’ (‘software’ matches the package-id and a ‘licensee’ currently is either a hostname or Domain\Username). This information can be pulled from the chocolatey*.ps1 scripts via inline C# … … …

Multiple Chocolatey Package Source Repositories

If something turns out to be working great, you’ll want it to scale with your needs. In order to be able to scale ‘company-wide’ it seemed to be a good idea to split-up our Chocolatey packages into multiple package repositories for the following reasons:

  • packages could be grouped logically
  • (office packages, developer packages, testing packages, …)

  • package visibility could be controlled by the activated package source repositories
  • (office hosts don’t even get the chance to install i.e. 20GB of VisualStudio)

  • favor package versions based on different feed-priorities for different teams/ user groups
  • (team ‘X’ is on a ‘slow’-feed that only contains VS2015 update I, not II because a certain VC compiler settings would be unwanted for Product X, team ‘Y’ is on a ‘fast’-feed with all the VisualStudio versions available)

We’ve came up with a ‘hierarchy’ that can be described as follows:

(technology)-[(department|team)*]-[(project)*]

I will devote a whole blogpost to this, so stay tuned 🙂

Chocolatey Support

What should I say? I’ve had a couple of talks with Rob Reynolds, Gary Evan Park, Rich, Kim, Pascal,… (yup, basically the whole Chocolatey team). If you need help or have any question – just get in contact via gitter, they’ll be happy to help.
I will also be hanging around there sometimes 😉

Grazer Linuxtage 2017

Just like almost every year, my friend Martin and I attended the “Grazer Linuxtage”, a free conference that annually takes place at FH Joanneum Graz. (#glt17)

We’ve more or less set-up the track for single board computers, consisting of a workshop on Friday, as well as an info-stand and a talk on Saturday.

Thanks to the 200+ people that visited our talk, hope you enjoyed it!

 

Our slides are available for download, feel free to contact us if you’ve got any questions concerning the discussed topics! [talk at Linuxtage website]

I’m building a PhotoBox

Finally I put up some time and started building a “PhotoBox” / “PhotoBooth”. Many thanks to my brother-in-law who’s skilled enough in handcraft to make this project possible without falling apart right away 🙂

The main goal of this project is to come up with an actual use case for one of my old Nikon D2x-cameras, that otherwise wouldn’t leave the shelf at all. Everything is build around the amazing SLR Booth android app – I’ve added a little Raspberry Pi spice (people using the booth should be able to download their pictures to their mobile phones for ~ 5 minutes after they’ve taken a picture).

Once I’m done with it and everything works as I expect it to, I’ll be sharing building instructions and the source-code for the RPi extension 🙂

What is the PhotoBox made of?

  • Wood
  • Screws
  • Metal Coupler Pieces
  • Nikon D2x
  • Pixel C
  • Raspberry Pi
  • Canon Selphy Printer
  • Cables

Why Working On Windows Is Fun Again

Hint: it’s because of PowerShell and Chocolatey! 🙂

I’ve just needed to setup IntelliJ with Scala-Plugin and JDK on a new machine:

choco install jdk8 -y
Chocolatey v0.10.3 Business
Installing the following packages:
jdk8 v8.0.121 [Approved]
Chocolatey installed 1/1 packages. 0 packages failed.
choco install intellijidea-community -y
Installing the following packages:
intellijidea-community v2017.1 [Approved]
Chocolatey installed 1/1 packages. 0 packages failed.

Merging XML with XSLT and PowerShell? – OK!

Combining XMl files from PowerShell – well, that’s pretty easy once you figured out how to work with

, but doing a correct & automatic merge turns out to be a quite challenging task.

Luckily there’s this: merge.xslt by (LGPL) by Oliver Becker – a XSL transformation ready accomplish this task in no time!

Let’s assume we’ve got two XML files.

FileA.xml:

<?xml version="1.0" encoding="utf-8"?>
<dict>
	<awesome id="21">Saxon</awesome>
	<awesome id="42">Chocolatey</awesome>
	
	<someweirdtag>
		I want candy.
	</someweirdtag>
</dict>
FileB.xml:

<?xml version="1.0" encoding="utf-8"?>
<dict>
	<awesome id="21">Coffee</awesome>
	<awesome id="42">Chocolatey</awesome>
	
	<someweirdtag>
		I'm afraid of catfish.
	</someweirdtag>
	<IhaveNoMemoryOfThisPlace/>
</dict>

What we want to accomplish is a merge of these two files – and there are several different possible outcomes:
If we just combine the two files, we’d want a result like this:

<?xml version="1.0" encoding="utf-8"?><dict>
	<awesome id="21">SaxonCoffee</awesome>
	<awesome id="42">Chocolatey</awesome>
	
	<someweirdtag>
		I want candy.
		I'm afraid of catfish.
	</someweirdtag>
	<IhaveNoMemoryOfThisPlace/>
</dict>

In the scenario I’ve been facing I needed to combine two files, favoring the second one – so all existing elements from FileA would be overridden by the elements of FileB.

<?xml version="1.0" encoding="utf-8"?><dict>
	<awesome id="21">Coffee</awesome>
	<awesome id="42">Chocolatey</awesome>
	
	<someweirdtag>
		I'm afraid of catfish.
	</someweirdtag>
	<IhaveNoMemoryOfThisPlace/>
</dict>

Adding the missing PowerShell script:

param(
[Parameter(Mandatory = $True)][string]$file1,
[Parameter(Mandatory = $True)][string]$file2,
[Parameter(Mandatory = $True)][string]$path
)

# using only abs paths .. just to be safe
$file1 = Join-Path $(Get-Location) $file1
$file2 = Join-Path $(Get-Location) $file2
$path = Join-Path $(Get-Location) $path

# awesome xsl stylesheet from Oliver Becker
# http://web.archive.org/web/20160502194427/http://www2.informatik.hu-berlin.de/~obecker/XSLT/merge/merge.xslt
$xsltfile = Join-Path $(Get-Location) "merge.xslt"

$XsltSettings = New-Object System.Xml.Xsl.XsltSettings
$XsltSettings.EnableDocumentFunction = 1

$xslt = New-Object System.Xml.Xsl.XslCompiledTransform;
$xslt.Load($xsltfile , $XsltSettings, $(New-Object System.Xml.XmlUrlResolver))

[System.Xml.Xsl.XsltArgumentList]$al = [System.Xml.Xsl.XsltArgumentList]::new()
$al.AddParam("with", "", $file2)
$al.AddParam("replace", "", "true")

[System.Xml.XmlWriter]$xmlwriter = [System.Xml.XmlWriter]::Create($path)
$xslt.Transform($file1, $al, $xmlwriter)

The eagle-eyed viewer spotted a caveat: yes, this does not run on Linux, there’s no

System.Xml.Xsl

in DotNetCore/PowerShell so far, but hopefully this will change! In the meantime, just swap the .NETish XSL code with our all-time-favorite Saxon!

java -jar saxon9he.jar .\FileA.xml .\merge.xslt with=FileB.xml replace=true

~ happy hacking!

Links:

Update: replaced $args in Code because of PowerShell 5.1 (thanks https://outofmemoryexception.wordpress.com/2016/08/05/powershell-5-1-14393/ )

Longterm Internet Connection Speedtest In Linux … With PowerShell ??

Something weird just happened: I’ve actually created the first PowerShell script that now regularly runs on my main Linux machine … the start of something new? …

I’ve been using PowerShell, well, A LOT at work lately – and the syntax is just so easy and straight forward – you gotta love it, even as a dedicated Linux user 😉

Script Speedtest.ps1

$outfile = "speedtest.csv"

[regex]$regexDl="Download\:\s(?'speed'\d+\.\d+)\s"
[regex]$regexUl="Upload\:\s(?'speed'\d+\.\d+)\s"

"Date;Time;Download;Upload;" | Out-File $outfile -Encoding utf8

while (1 -eq 1) {
    Write-Host -ForegroundColor Green "doing speedtest..."
    $meas = $(speedtest)
    $meas
    $download = $regexDl.Match($meas).Groups[1].Value.Replace(".",",")
    $upload = $regexUl.Match($meas).Groups[1].Value.Replace(".",",")

    "$(Get-Date -Format "yyyy-mm-dd;HH:mm:ss");$download;$upload;" | Out-File $outfile -Encoding utf8 -Append
    Start-Sleep -Seconds 300
}

The results will be shared in a couple of days when I’ve collected a significant amount of data!

Update: first results – upload speed pretty constant, download unstable!