How to Reduce the Hidden Factory in Software Development?

BY MARKUS SPRUNCK

This article describes the hidden factory in software development projects & maintenance and ways to reduce the hidden factory. The size of the hidden factory is different in organizations and teams, but something between 20 and 65 percent of total work capacity is not unusual.

No engineering technique and/or tool can bring more productivity than a reduction of the hidden factory. The key problem with the hidden factory is - that the efforts are not visible to management. Even many developers are not aware of the hidden factory.

Definition of the Hidden Factory

The term hidden factory comes from lean management approaches in real manufacturing - the hidden factory in software development can be defined as: The hidden factory in software engineering is the sum of non-value-added work in the

Software development process. It's the main source of increased maintenance & project efforts. The hidden factory is mainly waste of time, effort and other resources during non-value-added activities.

You may put each activity of the development process into one of the following three categories:

Value-added, e.g. implement the feature the business really needs and supporting artifacts
Non-value-added, e.g. working on defects, not needed reports or features that nobody needs
Non-value-added-mandatory, e.g. legal requirements for the product or development process

Figure 1: Charlie Chaplin is "hidden in the factory"; Standard YouTube License

Typical Sources of Waste

The main part of the hidden factory is rework due to errors caused in analysis, design, implementation, test and deployment phases:

Non-value-added Work in Analysis Phase

- Wrong/missing requirements

- Unnecessary requirements

- Changes after final reviews

- Feature creep

Non-value-added Work in Design Phase

- missing modeling standards (UML, ER, etc.)

- model inconsistencies

- design errors

- needed reverse engineering of existing code

Non-value-added Work in Implementation Phase

- defects and errors

- poor design (extensive refactoring needed)

- missing documentation

- poor development tools or missing integrated tools

Non-value-added Work in Test Phase

- missing test environments

- missing test data

- poor test cases and missing test concepts

- no test automation for regression tests

Non-value-added Work in Deployment Phase & Production

- needed hot fixes

- high number of incidents and problems

- failed changes into production

- increased effort due to missing automation (continuous delivery, DevOps, etc.)

How to Reduce the Hidden Factory?

There is no standard way to reduce the hidden factory; the only thing which is absolutely necessary is to make it visible. This means we have to analyze and quantify all avoidable efforts.

To improve efficiency and maintainability a systematic investigation of the hidden factory gives valuable information about improvable areas. Every organization will have own types of waste. When you know the root cause of the waste, the rest is common sense.