Agile development and pluralistic management: why Scrum fails

The Scrum methodology is often introduced because developers want it. On paper, it empowers them—if applied correctly—and developers need decision-making empowerment because development is a creative endeavor.

Of all the different types of people I've known, hackers and painters are among the most alike. What hackers and painters have in common is that they're both makers. Along with composers, architects, and writers, what hackers and painters are trying to do is make good things.
—Paul Graham

On the other hand, there is a shortage of highly skilled developers and companies think claiming to "do agile development" is a low cost option to make themselves more attractive to developers.

So, Scrum is being widely adopted and quickly becoming the de-facto standard methodology for software development.

A Forrester study from 2009 observed that 35% of organizations used agile. Actuation Consulting’s 2013 research shows that 73.68% have adopted agile to develop products. In comparison, only 12.5% of respondents use waterfall.

Unfortunately, the sad truth is that around 70% of agile adoptions fail. This happens because Scrum, and agile, are much more complex to adopt than they appear to be.

Ken Schwaber [...] suggests only 30% [of scrum-adopting companies] will become “excellent development organizations.” [...] Change experts like Harvard Professor John Kotter regularly say 70% of major change efforts fail.

High with the fumes of Dunning-Kruger, companies initially think adoption is easy enough. However, as Scrum is introduced, it becomes clear that the whole company has to embrace the agile principles, including sales and management, for the adoption to succeed in the long run, and many companies find they get more than the bargained for and go in denial. They start to blame Scrum for their own faults, and abandon it or start some weird chimera sometimes called "ScrumBut".

Instead, the correct way forward is to change the company management style and become a pluralistic organization, but it's a big step for most CEOs and management, mostly in terms of ego and self image, as well as fear of letting go and change resilience.

Stack Exchange's org chart

Stop thinking of the management team at the top of the organization. Start thinking of the software developers, the designers, the product managers, and the front line sales people as the top of the organization. The “management team” isn’t the “decision making” team. It’s a support function. You may want to call them administration instead of management, which will keep them from getting too big for their britches.
Joel Spolsky

There appear to be no shortcuts: until high-tech companies become pluralistic workplaces, they will not be good places to work for highly skilled, highly creative professionals such as software developers.

TL;DR:

  1. top-down management generally sucks for software development because building software is a creative act which requires diffuse responsibility to be done effectively;
  2. scrum (or, better, agile principles) are good tools to diffuse responsibility without creating anarchy;
  3. CEOs don't want to give away control and become a support figure to the company, which makes it impossible for their companies to become pluralistic, and so they keep on being ineffective.

Agile and scrum are great tools, but the context is much larger than a methodology.


Do you think it would be cool to work in a pluralistic company with Joel as a CEO? Come work at Stack Exchange, we are hiring full-stack developers anywhere in the world!

discuss this post on Hacker News


Hi, I'm Marco Cecconi. I am the founder of Intelligent Hack, developer, hacker, blogger, conference lecturer. Bio: ex Stack Overflow core team, ex Toptal EM.

Read more

Newest Posts

What can Stack Overflow learn from ChatGPT?

Stack Overflow could benefit from adopting a using conversational AI to provide specific answers

Read more
Fan mail

Multiple people with my name use my email address and I can read their email, chaos ensues!

Read more
Intelligent Trip

After years of building, our top-notch consultancy to help start-ups and scale-ups create great, scalable products, I think it is high time I added an update to how it is going and what's next for us.

Read more
Guest blog: Building, in partnership with communities by Shog9

A lesson in building communities by Stack Overflow's most prominent community manager emeritus, Shog9

Read more
Can you migrate a company from on-premise to remote-only?

Some lessons learned over the past 8 years of remote work in some of the best remote companies on the planet

Read more

Gleanings

And the Most Realistic Developer in Fiction is...
Julia Silge • Mar 28, 2017

We can say that Mr. Robot is having a moment. The main character was one of the top choices and thus is perhaps the most/least realistic/annoying/inspiring portrayal of what it’s like to be a computer programmer today.

Read more…