Friday, 13 October 2017

Help! I cannot complete my Pull Request in TFS!

A quick one today: if the Complete button is greyed out in your Pull Request UI, ask your TFS Administrator to start the TFS Job Agent.

image

The TFS Job Agent does many things – including handling Pull Requests completion.

Tuesday, 10 October 2017

Successfully handle disruptive changes with no downtime: the TFS 2015 example

It is something I’ve mentioned a few years ago but the question came out again during my presentation at x-celerate.de:

How should I handle a breaking change without service interruptions?

This is a brilliant question, and the best example I can give out is the TFS 2015 Upgrade that introduced support for renaming Team Projects.

If you don’t want to have downtime for your users the only mitigation is to introduce an intermediate migration layer which is going to be pouring data from the production stack and transform it into what you want.

The upside of this is that you are performing a very expensive and time-consuming operation out-of-band, so you can apply all the usual patterns for highly available application deployments.

The downside is that it is a costly operation, it could be compute, storage or something else but it will cost something out of it.

In my specific case I was able to perform a scheduled upgrade into the mandatory weekend window (yes, there was still a bit of downtime but it was due to the nature of the product and it was expected – you can overcome the hurdle if you are building your own product though) instead of having days of downtime due to the migration of data from a schema to another, at the cost of lots of storage space for the temporary tables and a dedicated server to run the tool.

Sunday, 8 October 2017

Re-release to an environment, don’t spin up a whole new deployment!

I know this happens on a regular basis – but it caught up my eye this morning as I am finishing up preparation for X-Celerate.de.

Let’s say your Release fails:

image

What many do is to actually spin up a whole new instance of the Release itself. While this works ok, you are missing out on something important: traceability.

In a sea of releases, with microservices and multiple moving pieces, how would you be able to trace back what happened during that failed release?

Why don’t you actually try to re-deploy the same failed bits instead?

image

Doing this provides you all the details about the previous failures, and it is going to be much easier to recall in case you might need to refer to the scenario in the future.

image

Oh in case you were wondering… it was all about my lab’s DNS server, which cached the Kudu website of an App Service I was deploying as 404 Smile but it is Sunday after all…

Thursday, 5 October 2017

If you don’t package your stuff you are doing it wrong!

Packages are a thing, exactly like containers are.

I mean – who wants to spend countless time in moving files, editing configuration files and the likes? Nobody, I know, but still so many people don’t take advantage of application packaging when it comes to deploying stuff!

Let’s take an average web application as an example. What is the reason that pushes you to actually move stuff from a certain folder (DLLs, .configs, etc.) to the target server, instead of packaging your application’s components and move these instead?

All you need to do is adding /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true to the MSBuild Arguments if you are using a traditional Build task.

image

If you are using .NET Core you just need to select the Zip Published Projects in the Publish task!

image

I just love the MSDeploy comeback. I am a huge supporter of this technology, because it makes life so much easier. It also has the side effect of enabling deployment to Azure in a snap, as it is one of the three supported delivery methods!

Let’s say you have a Build Definition and a Release Pipeline for this application. You want to deploy it to Azure – this is what you need to do:

imageWhat if you want to run the same application (providing it actually works on-premise as well and it does not specifically require any technology not available in your datacentre) on your own servers?

Firstly you’ll need to create a Deployment Group in VSTS – this can be done either by statically running the appropriate PowerShell script on your machines (interactively, via RDP) or dynamically with a bit of PowerShell or Azure if you are using IaaS. It is required because IaaS/on-premise machines in a Deployment Group will run an agent.

Then you can run whatever script you need to install the pre-requisites your application requires and configure all the settings, and eventually you can use the IIS Web App tasks to interact with IIS. Focusing on the IIS Web App Deploy task…

image

That Package is exactly the same package I used in the Azure deployment above. So you can easily have a Continuous Delivery pipeline on Azure and a different one (with the same cadence or a different one, your call) for on-premise, both starting from the same artifacts.

Containers are better – of course – but they require a minimum or ramp up or learning in order to actually implement them in a production environment. Moving to MSDeploy on the other hand is a matter of minutes at most, and it will provide a tangible improvement.

Friday, 29 September 2017

A few catches on customising the new Work Item form

If you use Team Foundation Server 2017 you already know this:

image

The new form is brilliant: it makes a much better use of the screen space, with a better UX in general.

But what if you have forms that were already using customised fields and a specific arrangement of controls?

image

The answer is that Microsoft uses a best-effort transformation system to automatically migrate your old form to the new, but for some reason I found myself in this situation – these two tabs won’t migrate to the new layout.

The new customisation model is brilliant – everything is now much cleaner and easier to use. All you need to do is add what you want to the <WebLayout> tag:

image

You can see that you now have Page, Group, Section as containers for controls, making life actually much easier when it comes to customisation. In my case I added two new Pages with the relevant control in there:

image

image

Of course you can customise the display layout by using the LayoutMode attribute. All the documentation is available here.

Wednesday, 20 September 2017

How to encrypt your Team Foundation Server data tier

For all sorts of reasons (including GDPR looming on you) you might feel the need to encrypt your Team Foundation Server databases. Proper encryption at rest.

I realised this is not a really well documented scenario, but it is surprisingly easy to achieve.

What you need to do is leverage SQL Server TDE (Transparent Data Encryption), which is out-of-the-box since SQL Server 2008 onwards. It acts at page level and it is transparent, with little overhead.

The process of enabling TDE is very well documented, and it is based off two keys (the master key and the encryption key) and a certificate. It is very straightforward if you have a single server as a data tier, off you go.

Now, this gets slightly more complicated if you have (like me Smile) AlwaysOn protecting your High Availability. Well, complicated if it is the first time you approach the topic.

Working with AlwaysOn requires:

  • On the Primary Replica - creating the master key, the certificate and the encryption key. Remember to backup the master key and the certificate.
  • On the Secondary Replica - creating the master key and the certificate. The certificate must be created from the backup of the Primary Replica!

After these two steps you can enable TDE on the database hosted on the Primary Replica, which then will propagate on the Secondary as per AlwaysOn schedule.

If your databases are already encrypted and you want to add them to an Availability Group you’ll need to do so manually – the wizard is not going to show encrypted databases to be added to the AG.

This SQLServerCentral.com article features a set of queries I found really helpful to get started.

A suggestion though: prepare a single query for the Primary Replica preparation, run it, prepare a single one for the Secondary Replica preparation, run it, and eventually encrypt from a separate query.

The reason why I say this is simple: if anything goes wrong before you encrypt the database you can easily drop the master key, the certificate or the encryption key and start again.

Eventually, remember that encryption for large databases can take a long time. During this time, the process might stop because of database size, so remember to check the logs as well so you can restart it if you need to.

Tuesday, 19 September 2017

Why Work Item rules are so important

I was on holiday for the last couple of weeks so I had only a limited coverage of what happened since the beginning of the month Smile but I could not miss the release of custom Work Item rules on VSTS.

Why such an emphasis on my end? Well, because custom rules on Work Items involved fiddling with xml files and command line tools or using the Process Template Editor in Visual Studio and a UI that is a bit tough.

image

image

It is something that any TFS administrator does on a regular basis though. Now rules in VSTS can now be defined in a web UI with a consistent experience and multiple sets of conditions and actions can be defined easily in the same page.

image

Also, this makes involving team managers in the definition of these rules extremely easy, as there is no Visual Studio, XML, command line involved anymore.

Wednesday, 30 August 2017

You might not know this: Service Execution History in VSTS

How would you know how many times your Service Endpoint is invoked, with what result and by who?

image

It is neatly after the Details and Roles tabs in the Service Endpoint list for each Service Endpoint Smile

Friday, 18 August 2017

Shadow code indexing jobs after upgrading to TFS 2017.2

Just a quick one – if you remove the Search Server before upgrading to TFS 2017.2, you might see failing Git_RepositoryCodeIndexing or TFVC_RepositoryCodeIndexing jobs raising a Microsoft.VisualStudio.Services.Search.Common.FeederException: Lots of files rejected by Elasticsearch, failing this job error.

The reason why this happens is because the extension is automatically enabled on the collections, triggering these jobs.

So check your extensions after the upgrade Smile

Thursday, 10 August 2017

Run a SonarQube analysis in VSTS for unsupported .NET Core projects

With some projects you might face this:

error MSB4062: The "IsTestFileByName" task could not be loaded from the assembly <path>\.sonarqube\bin\SonarQube.Integration.Tasks.dll.

It is a known issue I’m afraid, involving (among the others) .NET Standard.

There is a fairly straightforward workaround IMHO. Instead of using the Scanner for MSBuild as you would, use the CLI scanner that is now provided by VSTS:

sq0

This is enough to let the scanner do its job. This approach can bring a different issue – if you use branches to identify projects in SonarQube, or if you have dynamically set properties, and having a fixed, static properties file doesn’t really work.

Still, nothing really blocking. Do you see the PowerShell Script above? Smile

sq1

This is an example of what you can do – a bit rough, it just adds a line at the end of the file stating the branch to analyse. It can also be much cleaner, but still Smile

Remember that you can always manipulate files in the agent, and that’s what I do. Add whatever line you want with a script like this so that you have granular control in the same way as adding /d:… to the parameters in the regular MSBuild task.

Monday, 31 July 2017

My take on build pools optimisation for the larger deployments

If you have a large pool of Build Agents it can be easy to incur in a terrible headache: plenty of hardware resources to handle, capabilities, pools, queues, etc.

Bearing this in mind, having a single default pool is the last thing you want IMHO:

image

There are exceptions to this of course, like if you work on a single system (loosely defined, like a single product or a single suite of applications) or if you have a massive, horizontal team across the company.

Otherwise pulling all the resources together can be a bit of a nightmare, especially if company policy gets in the way – what if each development group/product team/etc. needs to provide hardware for their pools?

Breaking this down means you can create pools based on corporate organisation (build/division/team/whatever), on products (one pool per product or service) or on geography.

Performance should be taken into account in any case: you can add custom capabilities marking something special about your machines:

image

Do you need a CUDA-enabled build agent for some SDKs you are using? Add a capability. Is your codebase so legacy or massive that takes advantage of fast, NVMe SSDs? Add a capability. You get the gist of it after a while.

That becomes very nice, because with capabilities you can define your perfect build requirements when you trigger the build, and the system is going to choose the one that has all you need – saving you the hassle of manually finding what you need.

Maintaining these Build Agents is also important – that is why a Maintenance Job can be scheduled to clean up the _work folder in the agent:

image

This can have an impact on your pools – that is why you can specify that only a certain percentage is going to undergo the job at once. Everything is also audited, in case you need to track down things going south.

Wednesday, 26 July 2017

So many things I like in the new Release Editor!

Change is hard to swallow, it is the human nature and we cannot do anything about it Smile so like every change, the new Release Editor can be a surprise for some.

image

To be fair with you, I think it is a major step ahead, for a few reasons. Usability is on top of the pile of course, as I can have a high level overview of what my pipeline does without digging into the technical details of the process.

Then if you look at the Artifacts section, you will see the amount of sources you can choose from:

image

Being VSTS a truly interoperable DevOps platform spoils you for choice – I really appreciate the having Package Management in such a prominent place, because it enables all sorts of consumption scenarios for NuGet packages as a build output, including a cross-organisation open model.

Then on the Environments section, the templates provided cover lots of scenarios and not only with cloud technologies. One that is going to be really appreciated in hybrid DevOps situations is the IIS Website and SQL Database Deployment.

image

This template creates a two phase deployment that serves as a starting point for most on-premise deployments with IIS and SQL Server.

The Web App Deployment supports XML transformations and substitutions by default:

image

The data side of the story is really interesting IMHO as it employs DACPACs by default, together with a .sql file and inline SQL options:

image

I think it is clear I really like it Smile

Tuesday, 11 July 2017

Git, TFS and the Credential Manager

A colleague rang up saying he could not clone anything from the command line, but everything was fine in Visual Studio. All he got from PowerShell was an error stating he was not authorised to access the project.

He did not want to setup a PAT or SSH keys, and this behaviour was quite odd to say the least. There was also a VPN in the mix.

At the end of the day the easiest way to get around this was using the Windows Credential Manager:

image

Thursday, 29 June 2017

Some tips on Search Server for TFS 2017

Code Search is a brilliant feature based off Elastic Search. I wanted to summarise a few tips from my relatively limited experience with Search Server for TFS 2017, which adds Code Search to your instance.

The first thing to remember is that the size of the index can be quite large (up to 30% the size of a collection in the worst case) so plan for it in advance.

CPU and RAM should not be underestimated as well. Elastic Search can be a quite intensive process.

If you are not an expert on it (like me Smile) you want to use these script the TFS Product Team provided. They are straightforward and extremely useful. Also take a look at the documentation, it is very useful as well.

The indexing process can be quite tough on the TFS scheduler, so bear in mind this when you install it – otherwise you will see some jobs delayed.

This should not have an impact on the users (all the core jobs have higher priority anyway), but it is worth remembering that the indexing job will have a similar priority to the reporting jobs, so there is a risk of slowing reporting down.

Thursday, 22 June 2017

A few nuggets from using TFS/VSTS and SonarQube in your builds

The cool thing about SonarQube is that once it is set up it works immediately and it provides a lot of value for your teams.

After a while you will notice there are things that might be refined or improved in how you integrated the two tools, here are some I feel can be quite useful.


Bind variables between Team Build and SonarQube properties

I feel this is quite important – instead of manually entering Key, Project Name, Version, etc. you should be using your variables. Try to reduce manual input to a minimum.

Branch support

SonarQube supports branches with the sonar.branch property. This would create a separate SonarQube project you can use for comparison with other branches.

Analyse your solution once

Don’t be lazy and add just a task at the beginning and one at the bottom - you should scan one solution at the time and complete the analysis. This will solve the typical Duplicate project GUID warning you will get if you have multiple solutions in the same scan.

Exclude unnecessary files

It is so easy to add a sonar.exclusion pattern, do it to avoid scanning files you are not interested in.

Wednesday, 14 June 2017

How a (home)lab helps a TFS admin

I’ve always been a fan of homelabs. It is common knowledge that I am a huge advocate of virtualisation technologies, and pretty much all my machines feature at least Hyper-V running on them.

If you are not familiar with this, a homelab is a set of machines you run at home which simulate a proper enterprise environment. This does not mean a 42U cabinet full of massive servers, but even just a decently-sized workstation acting as VM host would do. The key here is enterprise, so ADDS, DNS, DHCP, the usual suspects indeed, plus your services.

What I am going to talk about is applicable to corporate labs as well, albeit this can be less fun Smile

So, if you are a TFS administrator, what are the advantages of a lab?

Testing upgrades!

Yes, test upgrades are one of the uses of a lab. But not just testing the upgrade itself, it also helps understanding how long an upgrade will take and what are the crucial areas you need to be aware of.

In an ideal world, you will have an exact copy of the production hardware so you might be able to have a very accurate forecast. This helps of course, but it will also hide what are the critical areas in your deployment.

Let’s take TFS 2017  – one of the most expensive steps is the migration of all the test results data in a collection to a different schema.

This is a very intensive operation, and having a lab where you know inside-out any finer detail about your hardware really helps when it comes to planning the proper upgrade, especially if you have a large deployment.

Without mentioning that in case of failure you are not breaking anybody’s day and you can work on your own schedule.

Also, you will find that sometimes you might need to experiment with settings that require a service interruption. The lab is yours, so you are not affecting anybody again and you can go straight to the solution when it comes to the production environment.

All of that sounds reasonable and maybe too simplicistic, but I saw too many instances where there was no lab and the only strategy was test-and-revert-if-fails, given that Team Foundation Server is “just a DB and IIS” (yeah…).

Definitely not something you want to see or hear, trust me Smile

Wednesday, 31 May 2017

What can you learn from The DevOps Handbook

I thought about reviewing The DevOps Handbook, but then I realised its real reference value. Books like this, with multiple layers of usage are really invaluable.


























This book is gold IMHO, and not just for its cover colour. But let’s rewind a bit.

I bought the book in January, I read the first few chapters then I forgot it at my parents’. Fast forward a few months, my brother comes visiting and he brings the book back, but it is then then left on the to read pile for a while.

Eventually I managed to find time to read it and here I am writing this post.
So why isn’t this a regular review? What did I realise, all of a sudden?

Well, books can be easily read in a few days, a couple of weeks tops. What happened is that each chapter I read was providing insights or mirrored situations I experienced during these months.

Each chapter can be cherry-picked and adapted to your situation, because it starts from a real scenario with real requirements and targets. You will notice patterns going through the book, all the concepts are there but applied in a tailored way to fit the problems one has to face.

I can see this book being a useful guide you might want to refer to when you are tackling a problem, just pick the chapter more akin to the approach you are using and you will feel guided.

Friday, 19 May 2017

Review - Professional Git

I am asked at least twice in three months about a good book on Git.

Despite the information available at https://git-scm.com/book is very comprehensive, there is value out of a published book to be consumed as a reference, especially when onboarding a new team member or when you finally want to get a firm grip on this (sometimes dreaded) Version Control System.

So I contacted Wrox to get a copy of Professional Git, which was published in December 2016 hence it is not only a reference book but also an updated reference book.


























You might wonder why is this so important to me. Well, it is fairly simple after all. Git became a mainstream tool a few years ago, but its history goes much further back. Documentation on its usage might be older, aimed at advanced users or fragmented, so it is important to have a book not only covering all the important topics, but crucially covering them from the right point of view.

Brent Laster did a brilliant job with this book. I felt the level of depth was perfect, neither too shallow nor too deep. Topics like Submodules – often troublesome for some and quite challenging anyway – are covered with a clarity that would make the essential information memorable.

The book has companion exercises and labs to keep you busy if you are a total newbie. If you are a bit more experienced it is still very valuable, putting down in plain language with straightforward samples even the trickiest of topics.

Just remember that the syntax used for the examples is UNIX-style – hence lots of ls! But aside from that I cannot refrain from suggesting it to who wants to start with Git and is looking for a comprehensive guide.

Tuesday, 9 May 2017

Error 404 when uploading the SonarQube Analysis Report from the Build

That can be something hard to catch if you don’t know where to look!

I spent some time on this issue – SonarQube’s End Analysis task fails, with either an unhandled exception (version 2.1.1 of the extension) or a 404 page in plain text in the log (with the older 2.0.0).

What was really odd was that the issue happened only on certain projects – some were fine, some others were failing, And it happened regardless of the build server used by TFS – whatever the agent, this randomness was there.

At the end of the day, don’t underestimate logs collected from the Web Server you are using: I found an error 404.13 in the logs, meaning the Analysis Report was exceeding the size limit for the upload.

Wednesday, 3 May 2017

What to do with WMI errors and Team Foundation Server

Last week I spent some time on trying to sort out WMI errors in a test environment. That was not fun, but at the end of the day there is something to learn out of it.
Everything starts with this set of errors with an AT-only installation:










Looking at the logs you can see it is pretty bad stuff:








I tried with the usual suspects (wmimgmt /verifyrepository, setting the involved machines as standalone hosts, etc), but they all run fine. One of the suggestions you can find around is to reinstall the IIS 6 Management Tools on the Application Tier:








Still no luck. I tried connecting with wmimgmt.msc and I got all sorts of errors. At the end of the day I temporarily reset the WMI repository on that machine and I then decided to move these test services to another VM. Why?

The WMI repository is not supposed to be rebuilt lightly. It should be the last resort, and I did not want to tie up this testing environment with a potentially problematic machine.
Don’t forget that the Application Tier is just a front-end for Team Foundation Server. You can replace it with another machine and scale out as needed.

Also, the errors above appear in both the Application Tier and Data Tier readiness check category for the AT-only install, despite they only apply to the AT. That is because the machine could not communicate with anything beyond itself (not even itself I would add), so it would report that the Data Tier isn’t reachable. Do not touch the database servers unless you really have to, and the logs are going to tell you if you have.

Tuesday, 18 April 2017

Quickly share query results with the Web Access

If you really like the Copy Query URL button because it opens a full-screen page with no other link to TFS or VSTS...

You would also know that the query link expires after 90 days. Too bad. Is there a way of getting the same behaviour (without taking into consideration any ACL-specific configuration) without using that link?

Well yes - just add &fullScreen=True to the Web Access URL:





Friday, 7 April 2017

“We don’t ship so often”: why? A reflection on Delivery hurdles.

The last Stack Overflow Developer Survey Results shows that the more a developer ships the happier (s)he is. 
Of course we see a huge amount of people checking in multiple times a day, but also a large amount of people checking in (so potentially building and deploying) much less than that.

So, looking at the other side of the medal: why aren’t you shipping often?

Reasons – as usual – are varied. There might be process constraints (certifications, etc.), hard requirements, but I’ve often seen a heavy reliance on older deployment procedures which are considered too expensive to be replaced by automation. Don’t touch what works, right?

Web applications are a stellar example of this. You might have the most complex web app in the world, but why should you manually move stuff around when you can pack everything in a MSDeploy package?

But that is for Azure and cloud technology and stuff!

Wrong answer! MSDeploy is around since 2009 and it is well supported on-premise as well! So why aren’t you using it for your existing application? It is, after all, the same concept Tomcat uses for its .war files.


This isn’t about throwing years of valuable content in the sink. It is often a matter of trying to split the larger problem into smaller components, and approaching different delivery vehicles. You can retain your existing application as-is, just replacing how you bring it into your production environments.

Sunday, 2 April 2017

How can I monitor my AlwaysOn synchronisation status?

As a Team Foundation Server administrator it is critical to have knowledge of all the components involved by your deployment, and SQL Server is the lion’s share (of course).

As you know I am a huge fan of SQL Server AlwaysOn, a really brilliant High Availability solution. I was wondering if there is a way of having an estimation of where the Database Engine is when you see the Synchronizing state in the AlwaysOn dashboard…








I found out there is a way, and it doesn’t even require any SQL at all. All you need to do is to add the Last Commit Time column on the Dashboard, so you will see the time of the last synchronised commit from the Primary Replica to the Secondary.














Of course it is not an ETA, but it gives a rough idea of how much work is left for the synchronisation.

During this state Team Foundation Server is still available because it relies on the Primary Replica, but remember to not perform any failover otherwise you are going to lose data! If it is a long synchronisation you are doing I strongly suggest to set the Failover Mode to Manual, downtime is always a better trade-off than data loss.

Tuesday, 21 March 2017

Move TFS databases with no downtime, thanks to SQL Server AlwaysOn

If you follow this blog or my Twitter feed you should know I am a massive fan of SQL Server AlwaysOn.

Recently I restored and moved some TFS databases around, and one of them remained on a temporary storage because of the massive size involved. After a while I managed to sort out the primary storage so I could move this database (and its Transaction Log) back to it.

This what I did, no warranties of course but it worked on my machines!

First of all, you need to be aware that you will have a limited availability during this period. It doesn't mean you are going to have an outage, but that you cannot rely on the Secondary Replica while you work on it. Why? Because you need to disable the Automatic Failover and make any Secondary non-readable:








Then suspend Data Movement from the Primary. This means your Primary Replica is not going to sync with the Secondary.















You will get your database to move in a non synchronised state.





Now note down your logical names for the files you need to move. Use these in the following query, the path in the FILENAME is going to be the new destination:











Run this on all servers. You might want to wait for the Secondary to be up-and-running, but don't forget to run it against the Primary too!





Copy all the files to the new destination, once done restart SQL Server on the Secondary:
















Now check that if Secondary is in a green state.









If the Secondary is green, resume Data Movement and after the status is Synchronised again perform a manual Failover so that the roles are swapped. Then perform all of the above on the new Secondary and you will be done.





Eventually, don't forget to re-enable any configuration you disabled before performing this!