My New Project Template

As of late, I’ve found myself hitting File > New Project ALOT.  It’s exciting to delve into some new packages and design paradigms, but I find myself taking on some of the same dependencies over and over again.  Some of my favorite that make it in virtually every time:

  • Serilog (link) – Structured logging
  • Serilog sinks, including, but not limited to File, Rolling File, MongoDB, SQL Server, RabbitMQ, and the Async wrapper
  • Mediatr (link) – In process messaging.  Jimmy Bogard, you are a legend.
  • Mediatr extensions to easily wire into Asp.Net Core.
  • NSwag for Api Documentation

I may remove one or two if the specific project demands it, but it beats reworking my Startup.cs every single time.

The folder structure is what threw me off and took the most time.  I tried to model it after most of the higher quality stuff I see on Github – things like IdentityServer4, and the Microsoft/AspNet repo.  I’ve gotten used to the ‘src’ folder with accompanying ‘docs’ and ‘samples’.  Achieving it was a pain, but I realized that creating Visual Studio Solution folders to be a close mirror to your actual structure helps to mentally organize it in your head.  A picture is worth a thousand words:

 

 

The top level has your .sln.

 

 

 

The next level in has historically been all of my csproj folders.  This time, I did a logical separation.  I may add one for messaging in the future as well.  Inside the logical folders (in the screenshot, CQRS) you will add your class libraries and projects.  Be careful though, Visual Studio can trick you here.

Add New Project will drop your .csproj into a normal flat list, but the physical location is right where you put it.  You have to use Add New Solution Folder to get a logical view in the Solution Explorer that matches your physical directory structure.  Bit of a nuisance, but it’s not so bad once you understand what’s going on.

Before Solution Folder:

 

 

(at this point, it’s actually physically located in one of the directories above)

 

 

 

 

 

 

 

After Adding Solution Folder via Right Click Solution > Add > New Solution Folder:

 

 

Just drag and drop your csproj into the solution folder, and check in the resulting changes to your sln file.

 

 

 

 

 

 

You could even do a totally different view in the Solution Explorer if you wanted, but my aim was to mirror most of the higher end stuff I see on GitHub in the .Net space.

I’ve thrown it up on Github:

https://github.com/Andrew-Brad/ApiKickstart

Hopefully it helps you save some time in your next project!  Happy coding!

1? Or…2?

I promise I’ve got some code samples coming down the pipe, but I’ve found another article which really caught my attention and I couldn’t resist the urge to provide my commentary.

(just in case you weren’t crystal clear on my development philosophy)

Thoughts on Agile Database Development

First off, I’m madly in love with Marten + Postgres.  The whole idea of a hybrid relational/object store at first prompted me to sound the alarm about one package or database being guilty of “trying to do too much”.  After working with it (albeit, on personal projects), boy was I wrong.  It’s really the best of both worlds – the friction-less object storage aspect (reminiscent of RavenDB or MongoDB – by design), combined with enough control to satisfy the majority of DBA’s I’ve worked with, makes for a truly new experience in the ORM world, where we Microsoft goons have been spoon-fed entity framework for many years.  I swear I spend more time configuring EF6 then I do actually writing code that matters.

The comparison of Postgres to MSSQL Server is out of scope here, but suffice to say that if you’re willing to take a dependency (normally something I would absolutely never accept), you’ll be richly rewarded.  Not only is IDocumentSession tremendously optimized for a given query or command (see my posts on CQRS), but the surrounding tooling for doing real life deployment tasks is tremendously useful, especially considering this thing is FOSS.  Schema comparison through the Marten command line takes so much work out of having to manage which SQL changes happened when, assuming you don’t have an expensive enterprisey tool to handle that stuff for you.  Even if you do, chances are it’s not linked up with your POCO’s, which is where all the interesting action happens.

Which brings me up to the next point – software design philosophy.

Quoted from Jeremy’s post –

“The” Database vs. Application Persistence

There are two basic development paradigms to how we think about databases as part of a software system:

  1. The database is the system and any other code is just a conduit to get data back and forth from the database and  its consumers

  2. The database is merely the state persistence subsystem of the application

Wow.  Nailed it.  I feel that there’s so much talk about DDD, stored procedures, ORMs, business logic, performance, ACID persistence, and which-team-owns-what-code that we never even stopped to ask ourselves, “am I a ‘1’ or a ‘2’?  The answer to that question shapes the direction of every system you build, and all systems that your team builds.

Alot of conflict I see arise in Software Development in my company comes from this aspect of writing code for the simple purpose of putting data in the database.  This is fine until you run into a non-trivial set of business rules and external integration points.  In my experience, one of the highest killers of scalability is over-reliance on a monolithic transactional store.  Our obsession to save a few bits of hard disk space has led to a colossal convolution and violation of separation of concerns.  I’m not allowed to pull a customer’s purchase history because there are 12 joins involved and the performance is too slow?

When did that become acceptable?

Now, Marten is not the golden hammer to this problem, but rather the design philosophy of the team that created it is explicitly aligned to more domain oriented thinking, versus the philosophy of “all hail the mighty column”.

His point about an Application Database hits home too.  I lost count of the number of times our dev team broke the database in our dev environment.  Being able to stand up a local copy for isolated testing (or intentional breaking) is unbelievably useful for anyone that gives a crap about proving the durability of their system.  I’m going to give Jeremy some more free linkage related to the shared database antipattern.

I added a Marten project to my Bootstrapper GitHub project to help capture some of my initial work with Marten.  It’s really for basic copy-paste reference, or a fast import of generally used CRUD methods that you’d want to hot drop into a File > New Project experience.  I still need to commit my VS 2017 migrations to master…

As an aside,

If you’re spending more time in SQL schema design meetings than you are with the domain experts, you’re doing it wrong!

But again, that statement depends on whether you’re a 1 or a 2…

I’ll leave the discussion regarding the ‘Vietnam of computer science‘ for another day.  Happy coding, friends!

 

Continuous Improvement

When I first started programming, I was under some illusion that one day, once you learned enough about the language, runtime, or design patterns, you could hammer out some brilliant lines of code, they would work almost magically, and you’d never need to touch it again!

Boy, was I wrong.

Turns out, there’s this word.  You all know it.

Refactoring.

We have a similar word in Lean/Six Sigma.  It’s called ‘Kaizen’.  In short, it means continuous improvement, but it has far reaching implications.  The word has its roots in Japanese culture & business (books that were written in the 80’s reference it), representing a philosophy of using data to constantly improve processes.  The major focus here is that it represents incremental improvement, not “big bang” improvement.  In the image below, the blue sections are slow, gradual improvements, intermingled with large jumps representing major advancements (red).

A more traditional visual of Kaizen may look something like this –

The key takeaway is that improvement happens gradually, and constantly.

A key tenet of Agile development (as argued by Dave Thomas of Agile fame) is that Agile software development is merely the process of making a small change, pausing to take a look at whether the changes have had a positive impact, and then course correcting if necessary.  Looks like the chart above doesn’t it?

A major component of this is refactoring.  In my opinion, every time a development team touches a codebase, they should leave the code in a better state than when they found it.  The goal here should be to improve readability, performance, organization, and overall flexibility of the code.  A Six Sigma-minded company pursues these opportunities to reduce waste, and only the worst run companies believe that waste is only financial.

Waste takes many forms – wasted time, effort, and talent, and all 3 of these are extremely relevant in software.

Wasted time results in delayed project deadlines, compressed testing, inter-personal frustrations, and a more rushed workflow in general, and rushing does not work in IT.  Sales people and high powered business folks may work better under tremendous pressure, but trust me, developers don’t.  Coding is a calculated and involved mental activity.

Wasted effort turns a 1 hour task into a 4 hour one.  It means you spent 3 hours copy pasting data from a spreadsheet into a custom JSON structure used by your app, only to find that in the end, it needed to be CSV format, and you could have just done a straight export from Excel.  Additionally, developers love efficiency hacks.  If wasted effort becomes a routine occurrence, they will rapidly become frustrated with their job, and will have reduced output and problem solving ability.  This makes for reduced team morale, and potentially increased turnover – something that HR teams may raise their eyebrow at you for.

Wasted talent is a truly hidden waste, and can silently kill large companies who are not prepared to respond.  A good friend of mine works extensively in finding and retaining the best talent for his business teams, and we’ve discussed this at length.  Hopefully I don’t need to justify this point, but if you think that all developers are worth the exact same amount, you have much more to learn about high quality software development. Steve Jobs probably could probably explain this better than I could.

Refactoring took me many years to fully appreciate, and I must admit, I’ve really come to love it.  It can feel like it can be its own art form sometimes.  Now, if you don’t have unit tests as insurance for all of your changes, you should probably go back to basics and get those up and running.  “Deploy with confidence” and all that.

There’s a ton of great material out there on refactoring, and I have yet to pick up my copy of the Fowler book on it. But, I’m keeping it simple.  Know your code smells, and make a series of small improvements in an effort to reduce waste.  If the design problem presents itself, maybe integrate a design pattern and assess the solution in a code review.  Over time, trust in the process that your codebase will improve over time.  The next developer will love you for it!

 

Water-Scrum-Fall

My team and I are in the midst of a transition into Agile-scrum.  It has been advertised to the rest of the company, and our executive management is pushing for it.

Thus far, we haven’t received any official process or reorganization to align with the vision, so we don’t have product owners, demo’s, retro’s, or discovery meetings.  We still do old-school waterfall requirements documents (which by the way, everyone wants to change just now while we’re firming up testing).  We joke that we’re in this purgatory of “water-scrum-fall”.

“That team has scrum meetings, so they must be Agile, right?”

I thought “water-scrum-fall” was a word we simply made up.  Turns out, it’s a real thing!

I recently watched a dev talk about scaling Agile and how to properly integrate Agile-scrum in the enterprise.  Sure enough, at 2:10:

It’s a real thing!!!

Wow.  To see our current project structure in this diagram blew me away.  I wasn’t crazy!  As a younger dev, I once got reprimanded for asking about requirements documents.  I now understand.  BDUP (big design up front) is basically a failure from the outset.  Without a product owner to be the primary gate of prioritizing work, everyone and their mother has input on what the feature “should look like”.

What this results in is a never-ending cycle of “sprints” (I use the term loosely in this context), with the work of sprint 5 modifying or undoing the work of sprint 2, so on and so forth.  Without a feature-driven delivery date and rapid push to production, water-scrum-fall is less productive than either scrum or waterfall.  If I chose a camp, I could just decide to never deviate from requirements, and save myself the time of a scrum meeting (which ends up evolving into just another 1-hour meeting where 12 people show up).  Or perhaps I would lock stakeholders into a meeting room and refuse to let them out until they’ve prioritized a backlog for the team to accomplish a deliverable feature.

Being in the middle, as the diagram shows, I actually find less productive than going all in on either methodology.

Jez mentions value stream mapping, and that is a critical skill that any Six Sigma belt will have, but that businesses outside of manufacturing generally don’t utilize.  Sure, they can define it, but they miss the point.  You need to understand who your customers are, internal and external, and what their inputs are.  This will be also different at a team level, so it’s important for all members to understand these concepts, just as they would be expected to understand a user story.

As he says, the lead time on a certain process may be perceived as 9 months, and people have become accustomed to ignoring it.  If that’s where 80% of the delay is, focus all of your energy and cutting the red tape and getting straight to value!  When you have centralized project management settings software deadlines from an armchair, you will absolutely get incorrect estimates and budget overage.  Agile software is improved bit by bit, constantly re-evaluated and measured.  This is also akin to how Six Sigma belts seek to achieve change in their organizations.  They never aim for a massive change in one fell swoop.  They expect a domino effect of smaller changes.  Instinctively, they understand that impactful change happens over time, incrementally.  One day you realize, “I don’t even need to do <task/feature/process> because of all the smaller improvements we’ve made to <other tasks/features/processes>.

Where then, do you begin with your improvements? The answer: low hanging fruit.

You may have heard of the 80/20 rule.  This is a staple in Six Sigma.  Understand that 80% of X may be caused by 20% of Y, or that 20% of your problems are impacting 80% of your performance hits.  It changes the way that you use data to drive action and continuous improvement.  I wouldn’t micro-optimize code without justification for a real benefit, so why would you think that having scrum meetings and calling your project phases ‘sprints’ will magically speed up development?

I hope it goes without saying that with my lean background and love of automated testing, I absolutely prefer an Agile scrum methodology, with a heavy focus on automated testability against a robust backend.  If you’re a team lead or scrum master who has some power and say over how you execute your agile processes, definitely give that video a watch.

The Evolutionary Architect

In the midst of tackling this guy:

book

I can’t even begin to express how encouraging and refreshing it is to have so many of my thoughts and concerns finally captured into written word and gaining momentum.  Things like domain boundaries, and the fact that data duplication is OK if done in the name of scalability and development team autonomy.

Not that I’m anywhere near Sam Newman’s experience and knowledge, mind you.  But my posts are fairly clear when it comes to my philosophy of design.  The database is not king, sharing a monolithic schema can and will kill your ability to scale in ALL of the following areas: product feature change, team delivery speed, reliability of the software, and uptake of new development talent.

“A good software design is easier to change than a bad software design.” – “Pragmatic” Dave Thomas.

(his blog)

One thing I truly admire about this book is Sam’s pragmatism.  He’s not trying to sell you microservices, he moreso does a thorough pro-con analysis.  The people that should appreciate this most are, indeed, software architects.

In chapter 2, The Evolutionary Architect, Sam goes through and does a deeper dive on what it means to be an architect, how we as a software development community have misunderstood the word over the years, and how a true definition is still up for grabs.  Honestly, I completely agree with him.  “Back in the day”, when I worked for Global Quality at a Fortune 500, I had the opportunity of a lifetime to study Six Sigma methodology with a true master of the craft.  This person not only knew the ins and outs of the methodology and the process, but they were responsible for managing a large global team.  It was under this person that I learned, by example, how an evolutionary leader can be both a master of a specific process, but also step back into a management role and empower their team to execute that process.  As team members (and myself as a junior member at the time), we can and will fail.  It is the architect’s (indeed – any manager) role to mitigate that failure, and manage the complexity involved.

It is an architect’s role to manage the complexity of a software product, not to increase it.

Unfortunately, since leaving that particular company, I have yet to meet another leader anywhere close to that magnitude of employee empowerment, mentorship, and expertise in both the “product” and the “people”.

So, back to Sam’s points (now that I’ve given you my background and why I agree), he states that the architect’s role is often that of a tech lead.  Based on my experience, alot of tech leads get less than 2 hours of coding per day, and are often caught up in meetings and company bureaucracy which prevents them from being directly or actively involved in the development. Sam states (and I agree) “More than any other role, architects can have a direct impact on quality of the systems built, on the working conditions of their colleagues, and on the organization’s ability to respond to change.”

This then, makes them such a strange hybrid of technical, but also leadership expertise.

Personally, I’ve seen both extremes – an architect who injects their opinion into code, without consulting the pragmatic ideas of the rest of the team (who in turn has to own the end result), and also the architect who is so hands off that their responsibility is configuring TFS and learning how to use Git so that they can tell the other team members to go Google it.

Neither of these scenarios capture the true essence of an architect – but Sam goes on to say we’ve actually borrowed terminology and not fully understood the impact – and that the role is very well defined in other industries, like engineering, where there is a specific, measurable goal.  By contrast, software engineering is less than a century old.

Trebuchet "is a" type of catapult - right?
Trebuchet “is a” type of catapult – right?

“Architects have a duty to ensure the system is habitable for developers, too”.  This is critical – tech turnover is still high.  Developers leave because they don’t like working with your codebase (yes, Mr. architect, you’re responsible for the overall quality of your codebase – go sit with your juniors now), or because benefits, culture, and environment is better at a different company.  In my experience, a company that invests wildly in the satisfaction of their employees retains better talent for longer.  Software is unique in the fact that you can invest in your developers with shiny tools and conferences, instead of being limited to “only” monetary compensation (like a sales team for example).

“If we are to ensure that the systems we create are habitable for our developers, then our architects need to understand the impact of their decisions.  At the very least, this means spending time with the team, and ideally it should mean that these developers actually spend time coding with the team too.”

This could be pair programming exercises, code reviews (you don’t get to complain about quality if you don’t put forth a concrete solution) or mentoring sessions.  If you’re an architect that only knows how to create stored procedures which end up creating multiple dependencies and breaking changes for more than one application, then you need to stop calling yourself an architect, and start doing your job – developers hate working in environments like this.  Stored procedures make for top-tier database performance, and the absolute lowest software agility (if misused) – since your dependencies cannot be managed from within your solution.  That “one guy” has to “remember” that “oh, when you change this sproc, these two applications will break”. Not fun.

 

Sam compares the architect to more of a town planner – they don’t get to decide which buildings go where, but they are actively involved in IT governance, and pragmatic decision making (read: data-driven) – i.e, they zone out areas where commercial and residential buildings will eventually go.

1538-2-sim-city-3000
Anyone remember SimCity?

A town planner does not have the power to add and remove buildings or real estate developers from those zones.  Often times, it’s developers who are on the cutting edge of new tools that can achieve various outputs, and they should be empowered to deliver on the desired quality.  If you’re dictating who is consuming which stored procedures, you’re a town planner who is calling up Wal-Mart and asking them to move in.  If your development team has assessed the risks, and has pragmatically agreed on Costco or Meier, you need to let them do their job.
I’m also a big fan of governance through code, as this hearkens back to my Six Sigma days of mistake-proofing a process.  This can open up a whole new area of discussion, such as how Resharper, or architectural styles like DDD, REST, and CQRS can enforce best practices (as defined by you) at a code level.  Another discussion for another time!


For any fans of mainstream house, you may be interested in Deadmau5’s new album – W:/2016ALBUM/ (not a typo!)

DI, IoC, and Others

tldr;

Dependency Inversion is all about getting rid of hard dependencies on your concrete types. When you new up an object, you take responsibility for its initialization, it’s lifetime, and take a hard dependency on its concrete implementation.  We want to eliminate the ‘new’ keyword from all around our code. This can be achieved with IoC Containers and Service Locators, which are slightly older and less featured than IoC Containers.  IoC containers exist specifically to manage object lifetime and initialization – they provide you a concrete type based on registration of an interface, thus ‘inverting’ the normal control flow of new’ing an object, then calling a method.  Instead, you explicitly declare your dependency against an interface in your constructor, then go on about normal business calling methods.  The inversion takes place because the actual object is instantiated and initialized in a dedicated object elsewhere in the app, thus following closer to Single Responsibility.


A colleague recently linked an article on dependency injection vs inversion, and how Service Locators compare to constructor injection, IoC, and the like.  It was a decent article, and I’d like to clarify some points which others found confusing.  Since this is all about the ‘D’ in ‘S.O.L.I.D.’, I’d like to start us off at square one to make sure we all start on even footing, especially if you’re new to the subject.


S.O.L.I.D.

Dependency Inversion.

Before I throw the Wikipedia definition at you, let’s look at some pseudo-code you’d find in a perfectly average controller method.

 

public IHttpActionResult GetAllPeople()
{
PeopleManager manager = new PeopleManager();

var allPeople = manager.GetAllPeople();

return new HttpOkObjectResult(people);
}

Even though it’s pseudo-code, the point is that you’ll typically find an instantiation of a service, a call into one or two of those service’s methods, then a return.

What’s the issue?

The issue is that you are not the only developer on the team, and the code inside PeopleManager will change.  Maybe some preconfiguration object will be required, maybe code might have to run on initialization in order to cache data inside the manager, perhaps some code will need to be disposed, prompting the use of a using statement.

If implementation code inside PeopleManager changes, will it break your controller code?  If the answer here is yes, we need to revisit our Single Responsibility principle!  Controllers are not for managing logic and excessive parsing and mapping.  Controllers should be the thinnest possible layer  between your app services.  They exist only to bind HTTP and/or route data to a service request of some sort.  They should keep your services HTTP ignorant and hand off your request, not manage the consistency of your services.

On the subject of consistency, what happens when you foreach through a  new List()?

Nothing!

This isn’t a technical question, it’s more of a philosophical one.  If you foreach through a new List, no Exception will be thrown.  There aren’t any elements inside, but you also don’t get a Null Reference Exception because of this.

The List, along with the overwhelming majority of modern .Net Types, initializes safely, keeping itself consistent for use.

This means that even though the backing array for List has no elements, and was not provided any information in the constructor, it did not null itself, and is rendered safe for use even in a brand new higher level object.

Objects are also responsible for keeping themselves in a consistent state (as much as they can given constraints and reasonable control over their internals).  That is to say the List is kept safe and consistent by exposing methods which express behaviour.  Add(), Contains(), Reverse() – all of these are clear with intent, do not violate SRP, and leave the object in a consistent state.

I say “reasonable control”, because external actors might interact with the List, and Add() a null value.  This might negatively impact external  code (attempting to access a null inside the foreach), but the List itself doesn’t blow up if nulls are passed to it.  Methods expose intent and behavior.  I can’t just reach into the List and set its backing array to null.

Code which uses the .Net List, takes on responsibility of initializing it properly, in the correct scope.

That’s all well and good, because List is a .Net type, and is a Type which is freely available to all of your architectural layers almost by definition, but extend the logic:

All of my controllers are responsible for initializing app services properly, in the correct scope.

Whoa!  Go ask your product owner what’s required to create a new <business entity here>.  A new customer?  Your product owner will tell you they need to agree that they are 18 or older, not that checkbox with id ‘chkBoxOver18’.checked() == true.  Same goes for your controllers.  They receive some bound data regarding the new customer details.  Should they be concerned whether the Customer Registration service requires a separate logging connection string?  Or that it should be used as a singleton?  Or that it’s unsafe to use it as a singleton?  Or that it has an array which is initialized as null, so if they use Magical Property A, they need to new up an array in Magical Property B? (I actually observed this in production code.)  Your controller’s responsiblity is, in loose talk, “I bind some data from HTTP, make sure it’s valid, and pass it off to one of our app services.  The rest is their problem.”  (A higher complexity enterprise app will generally use a request-response object type of pattern, but that’s out of scope for today.)

We’ve made one consistent definition, but the issue arises that in our second case, extending the definition violated SRP of our controllers.

Inversion of Control containers were born to alleviate the issue of instantiating complex objects.  They achieve this through a technique called Dependency Injection – which you can think of as constructor injection, though it’s not technically limited to constructors.

If your controller says, “I don’t care about telling the PeopleManager how to do its job.  My job is to let them know that I have the data they require to add a person.”

Here is how that is expressed:


public class PeopleController(){
private readonly PeopleManager manager;
public PeopleController(PeopleManager pm)
{
manager = pm;
}
public IHttpActionResult GetAllPeople()
{
var allPeople = manager.GetAllPeople();

return new HttpOkObjectResult(people);
}
}

We move PeopleManager to the constructor.  The controller is explicitly exposing its dependencies by moving it to the constructor.  Can a .Net FileStream exist without being given a file path or some sort of FileHandler?  No!

9 constructors - all require parameters.
9 constructors – all require parameters.

Likewise, your ‘PeopleController’ cannot exist without being given a reference to a PeopleManager to work against.  So where does this magical constructor parameter come from?

IoC containers handle object lifetime,initialization, and resolution.  In .Net Core, this is handled in Startup.cs.  Various registrations are made so that whenever an object asks for the Type, the IoC container manages an internal directory of which types are registered as which implementations.

startup

Transient means the resolved object will be instantiated just that one time, for that specific resolution.  You can see above that IDbFileStorage only requires some startup configuration code, but then is safe to hold in memory as a singleton.

The root of the Dependency Inversion Principle lies in the fact that Types should rely on abstractions, not concrete implementations.

This means that aside from all of this IoC stuff, the really good part of this principle only requires us to add a single letter!

Huh?


public class PeopleController(){
private readonly IPeopleManager manager;
public PeopleController(IPeopleManager pm)
{
manager = pm;
}
public IHttpActionResult GetAllPeople()
{
var allPeople = manager.GetAllPeople();

return new HttpOkObjectResult(people);
}
}

There!  Instead of PeopleManager, we code against the abstraction – IPeopleManager.  This has huge impact.  IoC is just one of the most common ways to achieve this (and it’s a soft requirement in .Net Core). Tomorrow, when an additional logging configuration object is required in the PeopleManager constructor, you don’t have to shotgun surgery all of your controller methods.  The change is confined to one place, and unforeseen breaks are easily fixed, without unexpected consequences in code which utilizes your manager.

Service Locators do something similar, but without constructor injection.  Conceptually, all they really do is expose a static reference which will give you the Type you ask for. I would submit that constructor injection is amazingly useful in writing transparent, expressive code, especially as workflows begin to traverse different services, or as services require other services and so on.

In the end, we’ve reduced the amount of code in our controller, relocated code to a place which is closer to its responsibility and intent, and made our end result significantly easier to read and refactor – and not to mention, test!  All of these contribute to a more Agile codebase.


What I’m listening to right now:

listening

Hangfire on .Net Core & Docker

This is going to be a lengthy one, but I did some setup for Hangfire to run in a Docker container (on my Ubuntu server at home)and I thought it’d be pretty exciting to share -given where we are in the .Net lifecycle/ecosystem.

What exactly are we setting up?

So, as part of my software infrastructure at home, I was in need of a job scheduler.  Not because I run a business, but because this is what I do for…um…fun.  I’m starting to have some disparate apps and API’s that are needing some long running, durable job handling, so I selected Hangfire based on their early adoption of Core.

I also completed my Ubuntu server build/reimage this past summer, and I was looking to be able to consistently “Dockerize” my apps, so that was a key learning experience I wanted to take away from this.

So here’s the stack I used to complete this whole thing:

  • Hangfire Job Scheduler
  • Docker – you’ll need the Toolbox if you’re developing on Windows/Mac
  • Hosted on my Server running Ubuntu 16.04 (but you can run the image on your local toolbox instance as a PoC.

The easiest place to start is getting Hangfire up and running.  I’ll  skip over my Postgres and Ubuntu setup, but that stuff is widely covered in other documentation.  I’ll have to assume you have a library for your job store that targets Core (I know MongoDB is dangerously close to finalizing theirs, and they have a Docker Image to boot!).  The one I used is shown below in my project.json.

So, spool up a brand new Asp.Net Core app; I made mine a Web Api with no security.   You can name it Hangfire.Web if you want to exactly follow along, but it really doesn’t matter, as long as you spot the areas where it would need to be changed.

In your program.cs, comment the IIS integration code.  We’ll be running Kestrel on a Linux VM via the Asp.Net Docker Image.

capture

Add your job store connection string to your appsettings.json.

Next up, tweaking your project.json.  I did a few things here, and I’ll post mine for your copy pasting pleasure.  The important parts are removing any IIS packages and pre/post publish scripts/tools.  By default, a new project will come with a couple of IIS publish scripts, and they will break your build/publish if you run only Kestrel in the container.


{
"dependencies": {
"Microsoft.NETCore.App": {
"version": "1.0.1",
"type": "platform"
},
"Microsoft.AspNetCore.Mvc": "1.0.1",
"Microsoft.AspNetCore.Routing": "1.0.1",
"Microsoft.AspNetCore.Server.Kestrel": "1.0.1",
"Microsoft.Extensions.Configuration.EnvironmentVariables": "1.0.0",
"Microsoft.Extensions.Configuration.FileExtensions": "1.0.0",
"Microsoft.Extensions.Configuration.Json": "1.0.0",
"Microsoft.Extensions.Logging": "1.0.0",
"Microsoft.Extensions.Logging.Console": "1.0.0",
"Microsoft.Extensions.Logging.Debug": "1.0.0",
"Microsoft.Extensions.Options.ConfigurationExtensions": "1.0.0",
"Hangfire": "1.6.6",
"Hangfire.PostgreSql.NetCore": "1.4.3",
"Serilog.Extensions.Logging": "1.2.0",
"Serilog.Sinks.Literate": "2.0.0",
"AB.FileStore.Impl.Postgres": "1.0.0",
"ConsoleApp1": "1.0.0"
},

"frameworks": {
"netcoreapp1.0": {
"imports": [
"dotnet5.6",
"portable-net45+win8"
]
}
},

"buildOptions": {
"emitEntryPoint": true,
"preserveCompilationContext": true
},

"runtimeOptions": {
"configProperties": {
"System.GC.Server": true
}
},

"publishOptions": {
"include": [
"wwwroot",
"**/*.cshtml",
"appsettings.json",
"web.config",
"Dockerfile"
]
}
}

You could honestly get rid of most of it for a bare-bones dependency build, but I left alot of defaults since I didn’t mind.
Next, Startup.cs:


public void ConfigureServices(IServiceCollection services)
{
// Add framework services.
services.AddMvc();

services.AddHangfire(options => options
.UseStorage(new PostgreSqlStorage(Configuration["PostgresJobStoreConnectionString"]))
.UseColouredConsoleLogProvider()
);

}

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IApplicationLifetime appLifetime)
{
loggerFactory.AddConsole(Configuration.GetSection("Logging"));
loggerFactory.AddDebug();

loggerFactory.AddSerilog();
// Ensure any buffered events are sent at shutdown
appLifetime.ApplicationStopped.Register(Log.CloseAndFlush);

app.UseMvc();
//Hangfire:
//GlobalConfiguration.Configuration
// .UseStorage(new PostgreSqlStorage(Configuration["PostgresJobStoreConnectionString"]));

app.UseHangfireServer();
app.UseHangfireDashboard("/hangfire", new DashboardOptions()
{
Authorization = new List() { new NoAuthFilter() },
StatsPollingInterval = 60000 //can't seem to find the UoM on github - would love to know if this is seconds or ms
}
);
}

And the NoAuth filter for Hangfire:


using Hangfire.Dashboard;
using Hangfire.Annotations;

public class NoAuthFilter : IDashboardAuthorizationFilter
{
public bool Authorize([NotNull] DashboardContext context)
{
return true;
}
}

 

Couple notes:

  • The call to PostgresStorage will depend on your job store.  At the time of writing I found a few Postgres packages out there, but this was the only one that built against .Net Core.
  • Serilog logging was configured, but is completely optional for you.  Feel free to remove it.
  • Why the Authorization and NoAuthFilter?  Hangfire, by default, authorizes its dashboard.  While I admire the philosophy of “secure by default”, it took me extra time to configure a workaround for deploying to a remote server that is still  in a protected environment, and I didn’t want to mess around with plugging in Authorization.  You’d only find that out after you deployed the Hangfire app.
  • Stats polling interval is totally up to you.  I used a (very) long interval since the job store wasn’t really doing anything.  To get the stats I need to consciously navigate to that web page, and when I do, real-time isn’t a critical feature for me.

At this point, you have everything you need to hit F5 and run your Hangfire instance on local.  Now would be a good time to double check your job stores work, because next we’re moving on to…

maxresdefault

DOCKER!

The Big Idea

The idea is, we want to grab the Docker Image for Asp.Net Core, build our source code into it, and be able to run a container from it anywhere.  As you’ll see, we can actually run it locally through Docker Toolbox, and then transfer that image directly to Ubuntu and run it from there!

We’re going to prep our custom image (based on the aspnet core Docker image here).  We do that by creating a Dockerfile, which is a DSL that will instruct Docker on how to layer the images together and merge in your DLL’s.

Note that the Docker for Visual Studio tooling is in preview now, and after experiencing some build issues using it, I chose to just command line my way through.  It’s easy, I promise.

First, create a new file simply called ‘Dockerfile’ (no file extension) in your src/Project folder:

capture
Your Dockerfile:


FROM microsoft/aspnetcore:latest
MAINTAINER Andrew
ARG source=.
WORKDIR /publish
EXPOSE 1000
COPY $source .
ENTRYPOINT ["dotnet", "Hangfire.Web.dll"]

Let’s take a look at what this means.  The ‘FROM’ directive tells Docker to pull an image from the Docker hub.  MAINTAINER is fully optional, and can be left out if you’re paranoid.  ARG, COPY, and WORKDIR work together to set the current folder as a variable, then reference the publish folder from that variable, copying in its contents (which will be your DLL’s in just a moment).  ENTRYPOINT is what Docker will call into once the host boots up the image.  You can call ‘dotnet Hangfire.Web.dll’ straight from your bin folders to double check.  Keep in mind the DLL name in ENTRYPOINT will be whatever you named your project.

To make life a bit harder, I decided to use a specific port via the EXPOSE directive.  I chose an arbitrary number, and wanted to be explicit in my host deployment port assignments.

See that publish folder from above?  We’re going to create that now.  I didn’t want to mess around with publish profiles and Visual Studio settings, so now is where we go into command line mode.  Go ahead and call up the Docker Quickstart terminal.  We can actually call into the dot net core CLI from there, so we’ll do that for brevity.

capture

Make sure kitematic is running your Linux VM.  Mine is going through Virtual Box.  I couldn’t tell you if the process is the same for Hyper-V driven Windows containers.  You might hang at the above screenshot if the Linux VM isn’t detected/running.

‘cd’ into that same project folder where the Dockerfile is and run a dotnet publish.  You can copy mine from here, which just says publish the Release configuration into a new folder called ‘publish’.

capture

cd ‘C:\Users\Andrew\Desktop\ProjectsToKeep\Hangfire\src\Hangfire.Web’

dotnet publish -c Release -o publish

Now, we have freshly built DLL’s.  We call into the Docker CLI which will pull the necessary image, and merge in that folder we referenced.

docker build ./publish -t hangfireweb

The -t argument is a tag.  It’s highly recommended to assign a tag as you can use that name directly in the CLI. If you get errors like “error parsing reference”, then it’s probably related to the tag.  I noticed some issues related to symbols and capital letters.

capture

Bam!  Our image is built!

I can prove it with this command:

docker images

capture

This next command will take a look at the image, and run a new container instance off of it.

docker run -it -d -e “ASPNETCORE_URLS=http://+:1000” -p 1000:1000 –name Hangfire hangfireweb

–name assigns the container a name so we can verify it once it’s live.

-d runs it in a background daemon.

-e will pass in environment variables.  These are variables passed into Docker when its constructing the container, and in this case, Asp.Net defaulted to port 80 (as it should) – but you’ll remember I explicitly instructed the container to only expose port 1000, so I need to also tell Asp.Net to listen on port 1000.  You can view other environment variables for each image on the Docker hub site or in the Toolbox.  Additionally, the -p argument maps the host port to the container port.  In this case, I opened up 1000 and mapped it to 1000.

You’ll get some output, and can confirm the container is up and running with this call:

docker ps

capture

(Keep in mind, if you restart the computer or otherwise stop the container, you can view all containers via:

docker ps -a

You can navigate to the Hangfire UI to make sure everything is dandy –

capture

That’s all!  To run the image from my Ubuntu box I just used docker save and docker load commands. (reference here.)  All you’re really doing is saving the image to a file, and loading it up from another server.  Nothing to it.  You can even keep the Toolbox instance running, spool up a second, and the 2 will compete over the job store.

Hopefully this was helpful!


I’ll sound off with a more urban, electronic/hip-hop fusion along the lines of Griz or Gramatik.  I found the album Brighter Future by Big Gigantic.  This is fun stuff, but will definitely be appreciated by a select few of you.

OSS vs. DIY

Fact:  Open source  software has absolutely exploded both in development and utilization this past decade, and it doesn’t seem to be slowing down.

For the subject of this post, “Open Source” means the dot net ecosystem of Nuget packages, as you would primarily find on Github.  Everyone already knows about the posterchild example of Linux (it runs an overwhelming majority of the world’s supercomputers), so if the Open Source revolution is news to you, welcome to the early 2000’s.

Back to Nuget.

Previously I’ve worked with some “old-school” folks who fear the fact that some mish mosh of DLL’s can be imported into your project, and all of a sudden, you’re liable for the code failing.  From old school developers who aren’t familiar with this world, I hear things like:

Why should we import this package?

What is the benefit?

Who is going to support it?

Why can’t we do it ourself?

All of these questions are rooted in the fact that the questioner has not delved into the code of the major packages out there.

By major, I mean any package within the top 20th percentile of commits and/or downloads.  These show extraordinary health, resilience, quality, and durability.

If a brilliant engineer at Google walked into your office, offered to show your dev team some of the best development tricks and secrets, leave you with code samples, and charge nothing, would you honestly say no?  If the SQL mastermind behind StackOverflow offered to visit to show you how their backend works with incredible performance and ease of coding, and charge nothing, would you honestly turn them away in the name of wanting your own custom ADO.Net code?

If you would honestly do it yourself rather take in the code of experts in your field – then your software is prone to bugs, poor performance, and will create massive waste in time and resources due to being rigid, and difficult to change and manage.

So, your status quo has been to “Code It Yourself”.  Bringing the logic in house is the way to go, right?  We write it, we own it, we maintain it.

Couple of logical flaws, here.  Let’s take a look.

Take a package such as RESTSharp.  Now being the type of developer who likes to stick fairly close to my own HTTP wrapping backend, I typically would recommend against taking a dependency on something like RESTSharp in enterprise projects – so there’s no bias here.   Take a look at their unit test suite:

restsharp

Now, if you think that developer hours are better spent “rolling your own solution”, instead of identifying potential open source libraries for a particular problem, I have some provocative questions for you.

Does this single package library have more unit tests than an entire application your team has developed?

Have you hired a Microsoft MVP to give your team code samples?

Do you believe that your development team is superior to the collective wisdom of thousands of developers across the world?

Are you building out and sharing native DLL’s (either internally or externally) for 12+ platforms?  How easy is it to integrate with your system? (SSIS doesn’t count – we’re talking business logic here).  How easily can you share code with your mobile developers for something like C# extensions?

Based on my conversations with folks that keep Github at a distance and their own monolithic codebase close to their chest, they fall short in each of these areas.  Microsoft MVP’s are required to contribute to the community by definition.  I didn’t link to Microsoft’s site because their CSS was screwed up at the time of writing and the MVP page was broken (lol).

Counterarguments to OSS come from a place of comfort and familiarity.  The sort of thing that drives “we’ve always done it this way”.

And I understand – continuous integration terrifies you.  Pushing to production terrifies you.  With good reason.  Every time you change the code something breaks.  Don’t you wish you could remove that fear?

Fact:  The fewer lines of code that exist in your application, the less opportunity there is for bugs, breaks, and showstopping errors.  When’s the last time a Hello World app broke? Finding open source packages which have a narrow, specific purpose, removes lines of code from your app and outsources common functionality into a peer-reviewed, proven and reliable codebase.  Less lines of code means less maintenance, and cleaner readability. (in the vast majority of cases)

But I don’t want to train developers on new language semantics for all these packages!

Another fallacy.  We used to parse like this:

 


public static T XmlDeserializeFromString<T>(this string objectData)
{
return (T)XmlDeserializeFromString(objectData, typeof(T));
}

public static object XmlDeserializeFromString(this string objectData, Type type)
{
var serializer = new XmlSerializer(type);
object result;

using (TextReader reader = new StringReader(objectData))
{
result = serializer.Deserialize(reader);
}

return result;

Now we parse like this:


JsonConvert.Deserialize<MyType>(input);

To prove my point about less lines of code – say you wrote that custom XML code. Your junior developers now have to:

  1. Remember to close the stream, or understand what the compiler does with using { } statements.
  2.  Pass in a Type as a parameter to a method, which is rarely used and is considered bad practice (the industry heavily uses generics)
  3. Address exceptions thrown if other XML conditions are not met (namespaces, prefixes,etc).
  4. Get in the habit of seeing a (T) hard type cast, another bad practice.

Developers import nuget packages to make their lives easier, and this means lessening the coding they have to do.  Open source packages seek out elegant code API’s which occur by capturing the natural language intent of the logic directly into the code.  This is a concept known as Literate Programming.  LINQ is the perfect example of this, but another open source example would be Shouldly.  If the Shouldly package doesn’t make sense to you right way, then shame on you for not unit testing!

1-7j-rape1td6t3ckazwbnyg

Jokes aside, less for loops and null checks mean clearer logical intent and higher code quality!

Now I’m sure a creative developer can find a way to break the call to JSON.Net, but I imagine it would be rather difficult, and you would most likely get a neat, descriptive little exception message telling you what’s wrong, versus some ambiguous TypeCast Exception.

Technology moves, people.  If your development team isn’t moving at a pace comparable to the technology curve as a whole, you will get left behind and create excessive waste in time and resources when the software breaks.  It’s remarkably hard to measure waste in these areas, but a big one is production support.  Having people up at all hours troubleshooting 4 nested for loops because one of your 15 variables happened to be null, is not only a total waste of time and talent, but steadily drops morale and kills the potential of your developers to do what you hired them to do:

Solve problems and innovate solutions for the business.

Let’s stop reinventing the wheel, and start adding more value.


I discovered a great new song by Calexico, a group that does a hybrid of Americana/Mexican music.  Check out Coyoacán.

There is bad code, and then there is…

…this gem:

public static void Do(ICollection people, string evaluationLogic)
{
for (int i = 0; i > people.ToList().Count; i++)
{
if (evaluationLogic == null) break;
if (evaluationLogic == string.Empty) people.ToList()[i].Name = "Bob";
if (evaluationLogic.Length >= 0) people.ToList()[i].Name = "Jim";
}
}

(mind the tab spacing – I’m still working out the kinks in some code formatters)
Psuedo-coded from actual production code.  In a site that was written this side of the 2000’s.  I will follow up this post with my commentary on hiring practices and some ways to improve getting good coding talent on your team, but suffice to say – whoever wrote this should not be writing production code.

Let me reiterate this – I actually found this live in production.

Where to even begin?  The Do() method is obviously a shell – but the point is you are given an ICollection – which has no indexers.  Some developers carry a preference to ToList() conversion, while some (like me) try to stick as closely to the least specific collection interface – which ends up being IEnumerable the vast majority of the time.  The advantage of coding against IEnumerable is it forces you to think of the target collection as an abstraction, allowing you to use the most functional and/or expressive LINQ statements to encapsulate your intent, without getting buried in nested for-loops.  Let’s call out the specifics:

  1. Repeated ToList() conversion.

Calling:

new List();

is valid C#.  It builds and runs.  It also does nothing for you as a developer.

Calling:

var content = new List();

instructs the runtime to assign a new collection object onto the heap, and store the reference to it in your given variable – here ‘content’.  The first snippet simply allocates an object, but does not capture a reference to it – so the GC will simply pick it up whenever it feels like.

When you feed a ToList() conversion into that first for-loop ‘i’ variable, you’re doing the exact same thing – allocating stuff onto the heap – accessing an int for a brief instant, but instantaneously releasing the reference, meaning your conversion was wasteful, and is needlessly stressing out the GC.

2. For vs ForEach.

The posted code seemed to utilize the poor ToList() conversion due to the need for the ‘i’ indexer – but when working with the collection interfaces, more than likely a foreach more than suffices.  The only reason not to is if you need to place a new object or change the reference of the object itself – with the foreach placeholder variable, you wouldn’t be able to do this.

3. Multiple redundant evaluations.

Notice the three if() checks?  Redundant evaluations.  The code I had assessed had mutually exclusive conditions, but was constantly rechecking them as part of normal loop flow.  An else or switch is everyday practice.

4. Break.

Code smell.  In something that isn’t algorithmic in nature (say, a performance optimized extension method, or just a highly performant loop), I find break to be awkward to use in everyday code.  If you’re breaking out of a for loop under a condition, then most of the time you can easily migrate to a LINQ statement that neatly encapsulates your intent.  Please don’t make me figure out your 10 lines of micro-optimized looping code (read: premature optimization), I have other things to do with my time.

2c96a9fa4c6b4c3cc0fe8322e13ae534

 

URN’s & URI’s, oh my!

Being more of a pure C# backend kind of guy, I’ve had to integrate with another SAML system recently.  I’ve actually never worked with SAML, so I dug into the spec and did the standard Googling.

As is not uncommon, I ended up on a wild goose chase of hyperlinks, after trying to find out what this meant:

"urn:oasis:names:tc:SAML:2.0:protocol"

It’s tucked inside standard SAML nodes (which is built on top of XML).  I figured I’d share with you what this whole “urn” moniker is all about.

All developers know what a URL is.  Most would call it a hyperlink or “web link” of some sort.  You’d be half right.  It stands for Uniform Resource Locator – and technically, it’s a subset of a URI – Uniform Resource Identifier.   A URI fits the following pattern (see RFC 3986):

uri

(grabbed from Wikipedia)

So our standardized definition would be “A compact sequence of characters that identifies an abstract or physical resource.”  The “physical or abstract” part is important – the target resource could be a physical file, or could return a representation of a logical piece of data.  Remember, to good .Net developers , XML and JSON are merely the representations of our objects – they are the means, not the ends.

So, a URL is a type of URI – specifically a URI that is located over a network.  A URL could be any of these:

https://app.pluralsight.com/library/

ftp://ftp.funet.fi/pub/standards/RFC/rfc959.txt

ldap://[2001:db8::7]/c=GB?objectClass?one

The first section is our network protocol (referenced as scheme), followed by host, port, path, and query fragments.

Being a visual person, the image made much more sense than delving through a spec somewhere trying to make sense of all the terminology.

 

I’ve always viewed the URI class in dot net as not adding much value – but after better understanding the URI structure, I have a whole new appreciation for it! Take a look at these:

Uri ftpUrl = new Uri("ftp://ftp.funet.fi/pub/standards/RFC/rfc959.txt");

Uri ldapUrl = new Uri("ldap://[2001:db8::7]/c=GB?objectClass?one");

Yields some pretty useful information automatically, without tedious parsing!

uri_vs

A URL is a type of URI, and, as it turns out, a URN is a type of URI! Uniform Resource Name is still a type of identifier, just intended to identify a hierarchical namespaces.

For example: urn:isbn:0451450523

A URN is still supposed to be a unique identifier, so the above line implies that the ISBN number refers to one book, and one book only.  So when it came to define the SAML protocol in a SAML message itself, it made sense to use an accepted standard instead of something custom.  OASIS can manage its own naming schema and convention under the “urn:OASIS” namespace.

As an aside, if you’re in web development, don’t go using Uri for parsing and doing tricky things with query strings. It wasn’t designed for that, and you’d be using the wrong tool for the job.

To get a little more “spec-y” – a resolution request can be made for any URN (of which these examples are) – but a URN resolver is responsible for taking that URN and translating it into a URL, or the resolvable location of that identifier.  This is somewhat abstract – but it’s quite simple really – a DNS server takes the URN of “https://app.pluralsight.com/library/” and resolves a physical network location to serve up that resource.  Subsequently, the web server accepts that URL and serves up the resource representation.  This is, in effect, a form of decoupling, which I find rather interesting.

It seems as if the deeper you get into web development, the farther back in time you end up. Indeed, this holds true for our industry as a whole.  REST API’s are my current specialty, and seeing so much similarity in terminology (“identifiers of resources”, “actions upon those resources”, “resource representations”) fascinated me when I first saw this.  It served to reinforce my commitment to the future of REST – as opposed to something new, custom and non-uniform.  REST is how the internet works – it makes sense to develop web services which minimize the deviation from natural HTTP constraints (ASP.Net session state is one such example – phenomenal idea when it came out – now pretty much everyone regrets it). I’ll be blogging about REST alot, but if you need another hyperlink to continue on your stream of curiosity, check out Roy Fielding’s work – any REST tutorial will reference him.