Writing code, a creative process, and agile software development

Writing code, a creative process, and agile software development

The care and feeding of software engineers (or, why engineers are grumpy) is a wonderful post writte by Nicholas Zakas. It states difficult interactions between developers and the rest of the world, and the misunderstandings on coding process nature, come from not recognizing writing code is a creative process. This post is rather long (8 pages, or 37 pages with the 200+ comments !), but it's definitely worth-reading.

This Nicholas post is divided in 2 parts:
* The first part (1.x) presents the strong relationships between coding and creativity, to point out current problems with developers come from coding not recognized as a creative process.
* The second part (2.x) gives hints to improve the current situation.

It's surprising to note that Nicholas does not mention agile methodologies, like SCRUM. Indeed, for various cases, an agile method could provide organizational improvements, inline with Nicholas hints. For example, for the case "why we [software engineers] say 'no'", SCRUM allows to tell the requested features could not be put into the current already fully booked sprint, but into the next one(s). Another example is that sprints enable to start coding, even if, contrary to other engineering fields, the specifications are not completely finished. So, in those 2 cases, SCRUM (or another agile method) could give an adequate answer enabling to organize developers/non-developers relationships in an acceptable way for both parties (and it gives means of action to developers, enabling them to be than mere laborers). So, again, it's surprising Nicholas Zakas has not at least dropped a word about agile methods.

Below is a summary of the original post, for readers in a hurry; note that the 1.x) titles, below, are mine, and the 2.x) titles are the original ones.

I have a theory. That theory is that software engineers see themselves very differently than those with whom they work.

1.1) Writing code is a truly creative process not recognized as such

There are a lot of companies that want short-order cooks, they want implementers and builders to march to a specific beat and stay in line. In fact, I’d say most companies want that, large and small.
And here’s the real crux of the problem: software engineers aren’t builders. Software engineers are creators. Building is what you do when you buy a piece of furniture from Ikea and get it home. The instructions are laid out and if you go step by step, you’ll get that comically small table you wanted. Creating is a different process, it’s birthing something without direction or instruction.
Software engineers don’t get into coding because they want someone to tell them what to do, they get into it because they discovered they could create something useful
In the triumvirate of software, product managers, designers, and software engineers, only the engineers are expected to turn off their creative minds and just produce. Software engineers aren’t dumb, they see this happening, and resentment starts to build (no pun intended). The engineers want to be part of that creative process and are denied. Thus you end up with the typical software engineer personality accented by grumpiness.
Both engineers and product managers tend to think, incorrectly, that product specifications or requirements are equivalent to the furniture manual from Ikea. In reality, these documents rarely contain enough information to build an actual thing. They’re usually just the starting point. And that presents a unique problem to the engineer.
In almost every other industry where things are built, it is expected that all requirements and details are agreed upon and finalized before building commences. Except in software. In software there’s “not enough time” to gather all the requirements ahead of time.

1.2) Writing code and creativity are linked, and fight the same ennemy: context switching (as opposed to the flow)

The enemy of any creator is context switching. Once you get into a deeply creative mode, “the flow” as some call it, being disturbed to shift focus to something else completely interrupts the process. Yes, writing code is a creative process. It’s both logical and creative at the same time. We aren’t simply writing code, we are crafting it.
...After all, as some people have told me, effort is effort. You just direct it where it needs to go like a cannon and fire. Of course, that’s not at all true. If you spend a lot of time on one task and then are asked to drop it to work on something else, you can’t very easily go back to the first task and pick up where you left off.
This is one of the top things that make engineers grumpy: constantly shifting priorities. If something is a number one priority on one day and something else is a number one priority on the next day, that means inevitable context switches must occur. Creative types don’t like being interrupted until they’re finished, which is why engineers are happy to continue coding until the wee hours of the morning just to complete what they’ve been working on. Interrupting the flow makes us less productive.
The frequency with which people above us change their minds is incredibly frustrating for software engineers.

1.3) Programming, as a creative process, implies difficulties to give time estimates

We have [software engineers] a tragic flaw and that flaw is that we overestimate our knowledge and abilities.
This flaw presents itself in a number of ways. The most frequent is in time estimates.
The problem is that, as creative people, software engineers fail to anticipate the problems they will encounter.
All of those add up to missed deadlines very quickly, but none do as much harm as the number one reason things don’t get completed on time: not factoring in time for learning. This goes directly back to our flaw. We think we already know how to complete the tasks we’re given yet very frequently they are things we’ve never done before. The time estimates reflect a state of perfect knowledge, where you have the Ikea manual and plow forward. In reality, many tasks are asking us to do things we’ve not done before.
Engineers who studied computer science in college are given a false sense of security in their classes.
So we come out of college thinking we know how to do everything when in fact we only know how to do what has already been done
Part of the problem is also our fragile egos. We get afraid that if we give an estimate that is “too long”, that people will think less of us. “Good engineers” should be able to work faster, they say, and so we acquiesce. I’ve always been amazed when an initial estimate is given on a project and a non-engineer comes back and says that it’s too long. First off, as I already mentioned, it’s probably actually too short due to our flaw. Second, how can a non-engineer know how long something will take to be built? And that leads to another issue.

1.4) A creative process, like writing code, is not just an hobby, it could be a job

There are few phrases that anger software engineers more than, “I used to code.” Whether it comes from a product manager, designer, or upper management, using this phrase in addition to a rationalization about why an engineer is wrong leads to nothing but disdain. If I were to ask LeBron James how much time he needs to prepare for a game, I’m sure he’d be amused if I disagreed because I played basketball in high school. Software engineers get the equivalent all the time.
The worst thing you can do for engineers is tell them you used to code.
But those who’ve never developed software professionally are better served to keep their coding hobby in their back pocket rather than using it as justification for anything in a business.
(To be fair, designers are also subject to this problem. Everyone is a visual design hobbyist because we all like pretty things. That doesn’t make everyone qualified to design something.)

1.5) Too many creative people, like too many software engineers, could harm the project progress

In some cases, adding a few more engineers will work. In most cases, adding more engineers only makes the problem worse. It is hard enough getting creative people to coordinate with each other, it gets more difficult as soon as you start adding more people in.
Having too many engineers on a project is a serious problem. Adding more engineers assumes that there are parallel tasks to be completed, but in reality, the number of parallel tasks on any one project is small and finite. When there are more engineers than can be used, engineering time ends up going away from developing and towards planning, synchronization, and coordination.

1.6) Software engineers write code usually to affect people’s lives

That’s really what drives software engineers more than anything else: the idea that people we don’t even know will be affected by our work. Whether you’re working on a web site visited by millions each day or you’re working on a point-of-sale system for restaurants, the knowledge that we’re affecting people’s lives is a powerful driver.
When there are delays due to people changing their minds, we get very grumpy. Insanely grumpy. Our goal of getting our work in front of people has been deferred, and that’s demoralizing.
Engineers don’t hate hard work or long hours; we hate when it doesn’t pay off.

2.1) What thanks?

As a software engineer, our jobs operate on very different timelines than others.
Yet, you’ll rarely find software engineers complaining about long hours or being woken up because of a production issue. The software is our baby, and we like to care for it as such. That means if it needs feeding in the middle of the night, we do it. If it needs extra care over the weekend, we do that too, all with a smile because our creation is growing.
Engineers are especially happy when they’re able to check in the last bits of code for a task. I’ve never seen engineers so jovial as when they send out an email saying a task is complete and ready for review. Yet that mood is quickly dashed when in the next ten minutes, bugs start getting filed against their newly-created baby.
All you really want is to take a moment to sit back and admire your work for a bit. Maybe have someone say, “good job.” And what response do we get? Bugs.

2.2) Why we say “no”

Keep in mind all of these [common reasons why engineers say no] except for the last one have to do with the engineer meeting a deadline to get the project out the door. We want the tasks to be completed, and the only way that happens is if the tasks don’t change while we’re working on them. When they do change, that’s when we get really grumpy and that’s when the “no” comes flying out of our mouths before you even finish your sentence.

2.3) Care and feeding

What interests us is coding and creating. When we can do that in a healthy environment, we’ll remain happy for a very long time.
So how do you create a healthy environment for engineers?

2.4) Work cross-functionally

Software engineers are creative, just like product managers and designers, and so you should work to include them in the creative process. Engineers are tremendous assets in brainstorming sessions and when reviewing initial designs. Give every engineer the opportunity to meet with the ideation team and work directly with them (not necessarily all at the same time). In short, inject the engineer into the creative process earlier. No engineer likes getting specs and designs thrown over the wall without understanding them.
What’s more, engineers are frequently way ahead in terms of knowledge of what is possible. If you consider front end engineers, we have knowledge about what browsers are capable of long before product managers and designers do.
So, invite engineers into the creative process early on. Let them give you feedback and provide information about what is possible. The less it feels like we’re being dictated to, the more likely we are to listen and happily go about our jobs. And that only really happens if we feel like we contributed to the creation of this thing.

2.5) Make a creative space

There’s a reason why hack days and hack weeks are so popular – it’s because this is the creative outlet that engineers need to refuel and rediscover their love of code.
Keep in mind that engineers aren’t special in this regard. Everyone needs time to be creative. In my experience, however, product managers and designers tend to get that far more frequently. There are offsites for management and design summits for designers, yet engineers tend to get left out.
You can also light the fire of creativity by sending engineers to conferences so that they can keep their skills up to date. Allow engineers to buy books that contribute to their knowledge on the company’s dime. Permit engineers to express their ideas about the projects they’re working on. Google famously gives engineers 20% of their time to pursue side projects. All of this can go a long way towards creating a great relationship with your engineers.

2.6) Encourage time off

With the amount of hours and mental exercises we do on a regular basis, engineers need to take breaks. Unfortunately, that’s also something we’re not very good at scheduling.
Engineer burnout is unique because we are used to powering through it. When burnout gets bad enough, we leave, looking for relief. What’s more, engineers will probably never tell you that they’re approaching that point; we’re too proud for that. On my last team, I told the engineers that the first time they felt frustrated, to come and talk to me. I didn’t want them to wait until it got so big that the only way they could escape was by leaving. I didn’t want them to leave, I wanted them to be happy, and the only way I could help is if I knew they were starting to not be happy.

2.7) Let ‘em code

As ironic as it may sound, a lot of companies hire software engineers and then don’t let them actually code. Instead, their days are filled with useless meetings that inhibit productivity. In general, software engineers are most productive when they can code for at least four hours straight without interruption.
It’s difficult to get into a good flow while coding if you know you have a meeting coming up in an hour or two hours, that’s always in the back of your mind while coding. It is amazingly unproductive to code for an hour, stop for an hour, code for an hour, stop for an hour, etc.
The software engineer brain has to switch into a good mode for coding in that switch takes time.
Make sure that your engineers have, every single day, at least four hours of uninterrupted time to code.
Also, take at least one day a week to have no meetings.

2.8) Express appreciation

This is something that can be done immediately and is completely effective. I mentioned earlier the frustration of toiling away to finish a task, only to be met with bugs filed against it. We engineers rarely get a chance to sit back and admire our work, let alone get a pat on the back from someone else.
Feeling appreciated is important to software engineers because most of the feedback we get is negative, in the form of bugs, production issues, and the like. A little bit of positive feedback makes the rest all that much more tolerable.
Every product is a success or failure because of all three groups, no one group could do it alone. Make sure your company always recognizes the team as a whole and not just one particular part.

*) Conclusion

We software engineers are an interesting bunch. There’s a definite personality that comes along with us, and we really do want to make the best thing possible. If you stop treating us like short-order cooks and start treating us like part of the creative process, you are likely to get much farther, faster than you would otherwise. The teams on which I’ve worked have all had varying degrees of friction caused by not understanding the engineers’ mindset and what drives them. It’s my sincere hope that this article will lead to better communication between engineers and those that they work with. It’s really not that hard. We’re all just people who want to feel like a part of the solution rather than a worker bee.

PS: for French readers, the book Ingénierie logicielle : vive l'artisanat ! defends the impossibility to industrialize and fully automate the coding process. I don't know if this book links explicitly coding and creativity, but it's not that far away as it presents software engineering as a crafting activity.

Posted at 12:18AM Oct 31, 2012 by Dominique De Vito in Developer  | 

View the original article here