Automation and context of use

The Heyns report on Lethal Autonomous Robotics makes a distinction between autonomous and automatic.

“Autonomous” needs to be distinguished from “automatic” or “automated.” Automatic systems, such as household appliances, operate within a structured and predictable environment. Autonomous systems can function in an open environment, under unstructured and dynamic circumstances. As  such their actions (like those of humans) may ultimately be unpredictable, especially in situations as chaotic as  armed conflict, and even more so when they interact with other autonomous systems.

Good point to make. The concept of 'predictability' for autonomous systems has problems when the environment introduces complexity - even for a simple device.


Here is a simple example of simple automation that did not capture the context of use.

Manual tap and soap dispenser.

Automatic tap (sensor under outlet turns on water flow).

See the problem? You reach across for the soap and soak your sleeve.

So - Human Centred Design is necessary even for the simplest of automatic systems. Context is just about everything.

Human-Centred Management - a case for standards?

This post is a follow-up to a debate at the IEHF Conference led by Dr Scott Steedman CBE, Director of Standards, BSI. That background has not been added yet, so the post may not be clear as it stands.

 We 'know what good looks like' as regards the human-centred management of people in enterprises. This note gives some pointers to that literature, with an emphasis on Pfeffer and Sutton's work on Evidence Based Management. A good summary can be found in the Happy Manifesto by Henry Stewart. The first chapter is called "Enable People to Work at Their Best". Perhaps using this knowledge to produce an inspriational standard would help the cause.
We need to promote what ought to be commonsense because it is overwhelmed by technocratic command and control thinking and an obsession with 'leadership'. The zillions of Something Management System standards promote the mechanistic management of things. Whilst this might be useful, on the basis of 'what gets measured gets done', such mechanistic procedures exacerbate some of the flaws in our society. A human-centred approach needs to be promoted to at least restore the balance. Fortunately the wherewithal to do this in a 'third generation' way have already been developed.

O'Reilly and Pfeffer have contrasted conventional strategy with values based strategy as follows:

[Based on: Hidden Value: How Great Companies Achieve Extraordinary Results With Ordinary People (Harvard Business School Press)]. I have had to deliver Value Plans working for BAE Systems - challenging and genuinely useful in my experience.

In 'The human equation: building profits by putting people first' Pfeffer has shown that a human-centred approach yields long-term business benefit.

We are overdue a paradigm change in the approach to people and safety. The new view of system safety has been well-developed by Woods, Dekker and others. 'How Complex Systems Fail' (pdf) would be a good starting point. The new paradigm has not taken hold (yet). Steven Shorrock has just written a terrific blog post on why this may be. Perhaps a standard would help.

There is over sixty years of literature and practice on Socio-Technical Systems - the conceptual foundation of ergonomics and human-centred approaches. A pointer to that literature can be found here.  In recent years, John Seddon's proprietary Vanguard implementation of systems thinking has found success in the UK public sector, producing benefits considerably in excess of a 20% target.

I will conclude with Pfeffer and Sutton "The single best diagnostic to see if an organization is innovating, learning, and capable of turning knowledge into action is 'What happens when they make a mistake?' "

Air Traveller User Experience (UX)

Air travellers are faced with conflicting stereotypes for document scanners; face up or face down. The check-in machine shown here expects my passport face-up.

The e-passport reader expects it face-down (which matches my expectation). This article says the future for boarding card readers is face up. Glasgow Airport has just installed face-down readers. It is clearly going to be a confusing mess for the next decade. Not life-threatening, but  along with the security theatre, a signifier of the clueless authoritarianism that lurks behind the functionalist aesthetic.

A collection of recently-collected confusing iconography above (not air travel, but while travelling). The first sign does NOT mean that you are safe from flames in the lift. It is very unclear what the sign adds to the text in the second one. The bottom indicator was clear to the designer, I'm sure..

The picture above is from the Hamburg Metro at the airport. A true gem. To go to the city centre, you press button 3. Not that button 3 - the one on the screen.

UX is about more than just functionalism. Going through Heathrow, I was delighted to see this picture of Herne the Hunter.

The celebration of local mythology is to be welcomed , but does it have to be so functionalist? A more evocative image is this one:

The UX of air travel is affected by the sense of place. For British airports, it is adversely affected by a complete lack of any sense of place from a combination of soulless functionalism and relentless mercantilism. Glasgow Airport was (properly) designed by Basil Spence, who ".. wanted a design which helped the traveller to feel the adventure of flying from this particular airport”. Well, the feel of adventure has gone, and the design has been buried in extensions. It is still possible to see the back of the original terminal.

The good news is that Wetherspoons understand a sense of place. They have put up a poster to Spence and provided a place where you can appreciate the  canopy (originally outside the building of course).

Is 'autonomy' a helpful aim for 'unmanned' platforms?

Building 'autonomous' platforms sounds exciting as an engineering challenge. This post suggests that the concept may be sufficiently flawed that it takes well-intentioned technical effort down a blind alley, and that a somewhat less-exciting conceptual framework may end up supporting greater technical advance.

Chapter 3 of Vincenti's classic book "What Engineers Know and How They Know It" is on Flying-Quality Specifications. The concepts of stability and control had to be re-thought. Essentially stability had been seen as a property of the aircraft on its own. This concept had to change to provide the pilot with adequate control. Flying qualities emerged as a concept that related to the aircraft-pilot system. I would give a better description, but someone hasn't returned my copy of the book. Changing the underlying concepts took a good decade of experimentation and pilot-designer interaction. My concern is that 'autonomy' as currently defined will hold back progress in the way that 'stability' did in the 1920's.  The 2010 version of CAP722 (Unmanned Aircraft System Operations in UK Airspace – Guidance) is used as the reference on current thinking on 'autonomy'.

The advantage we have for 'autonomy' over stability in the 1920's is that there is a good body of work on human-automation interaction, supervisory control etc.going back sixty years that can be used. There is well-established work that can elaborate human-automation interaction beyond a simple 'autonomous' label, or a 'semi-autonomous' label (reminders of 'slightly pregnant' as a concept). For example,

• Tom Sheridan defined five generic supervisory of functions planning, teaching (or programming the computer), monitoring, intervening and learning. These functions operate within three nested control loops.
• The Bonner-Taylor PACT framework  for pilot authorisation  and control of tasks can be used to describe operation in various modes.
• Work by John Reising , Terry Emerson and others developed design principles and approaches to Human-Electronic teamwork, using, inter alia, Asimov's Laws of Robotics.
• Recent work by Anderson et al on a constraint-based approach to UGV semi-autonomous control.

CAP722 (3.6.1) requires an overseeing autonomous management system. This has echoes of the 'executive' function at the heart of the Pilot's Associate programme. It is my recollection that the name and function of the executive kept changing, and perhaps proved too difficult to implement. A more feasible solution would be a number of agents assisting the human operator. It is not obvious why CAA guidance precludes such an option.

CAP722 (3.5.1) states: 'The autonomy concept encompasses systems ranging in capability from those that can operate without human control or direct oversight (“fully autonomous”), through “semi-autonomous” systems that are subordinate to a certain level of human authority, to systems that simply provide timely advice and leave the human to make all the decisions and execute the appropriate actions'. 'Full Autonomy' is thus a self-defining no control zone (Grote). As a pre-requisite for such a zone, the transfer of responsibility from the operator at the sharp end to the relevant authority (e.g. the Design Authority, the Type Certificate Holder, the IPT Leader) needs to be clearly signalled to all concerned. The 2010 version of CAP 722 seems to leave responsibility at the sharp end, and the inevitable accusations of 'operator error' when things go wrong.

I  leave the last words to Adm. Rickover:
"Responsibility is a unique concept: It can only reside and inhere in a single individual.  You may share it with others but your portion is not diminished.  You may delegate it but it is still with you.  Even if you do not recognise it or admit its presence, you cannot escape it.  If responsibility is rightfully yours, no evasion, or ignorance, or passing the blame can shift the burden to someone else.  Unless you can point your finger at the man responsible when something goes wrong then you never had anyone really responsible."

Update: Project ORCHID seems to have the right approach, talking about degree of autonomy required for tasks, and providing digital assistants. Also, please note the agent based approach.

Update: It's nice to be ahead of Dangerroom. "The Pentagon doesn't trust its own robots". Problems of autonomy!

Cognitive anti-patterns - more inputs

Don Norman's book “Things that make us smart” has Grudin’s Law: When those who benefit are not those who do the work, then the technology is likely to fail, or at least be subverted.
Amalberti's human error self-fulfilling prophecy: by regarding the human as a risk factor and delegating all safety-critical functions to technology as  the presumed safety factor, the human is actually turned into  a risk factor.
Gary Klein, Dave Snowden and Chew Lock Pin have listed 'useless advice' regarding anticipatory thinking. 'Useless advice' is pretty spot-on for anti-patterns. The useless advice is :

• Gather more data.
• Use information technology to help analyze the data.
• Reduce judgment biases.
• Encourage people to keep an open mind.
• Appoint “devil’s advocates” to challenge thinking.
• Encourage vigilance.

The 'devil's advocate' refers to a specific challenging role, rather than an independent overview role.  'Encouraging vigilance' is about vigilance not being a substitute for expertise, as opposed to mindfulness training.

Robert Hoffman provides some laws about Complex and Cognitive Systems (CACS). The laws are not quite patterns/anti-patterns, but look capable of being worked into that framework. Woods and Hollnagel have developed them into patterns for Joint Cognitive Systems. A number of the laws relate to 'integration work'. The following seem relevant:
The Penny Foolish Law: Any focus on short-term cost considerations always comes with a hefty price down the road, that weighs much more heavily on the
shoulders of the users than on the shoulders of project managers.
The Cognitive Vacuum Law: When working as a part of a CACS, people will perceive patterns and derive understandings and explanations, and these are not
necessarily either veridical or faithful to the intentions of the designers.  [bsj i.e. design intent needs to be explicit.]
Mr. Weasley’s Law: Humans should be supported in rapidly achieving a veridical and useful understanding of the “intent” and “stance” of the machines. Mr. Weasley states in the Harry Potter series, “Never trust anything that can think for itself if you can’t see where it keeps its brain.”
The Law of Stretched Systems: CACSs are always stretched to their limits of performance and adaptability. Interventions will always increase the tempo
and intensity of activity.
Rasmussen’s Law: In cognitive work within a CACS, people do not conduct tasks, they engage in context-sensitive, knowledge-driven choice among action
sequence alternatives. [bsj This links to Amalberti's 'ecological risk management'.]
Dilbert's Law: A human will not cooperate, or will not cooperate well with another agent if it is assumed that the other agent is not competent. 
Law of Coordinative Entropy: Coordination costs, continuously. The success of new technology depends on how the design affects the ability to manage the costs of coordinating activity and maintaining or repairing common ground.
Law of Systems as Surrogates: Technology refl ects the stances, agendas, and goals of those who design and deploy the technology. Designs, in turn, refl ect the models and assumptions of distant parties about the actual diffi culties in real operations. For this reason, design intent is usually far removed from the actual conditions in which technology is used, leading to costly gaps between these models of work and the “real work.”
The Law of the Kludge: Work systems always require workarounds, with resultant kludges that attempt to bridge the gap between the original design objectives and current realities or to reconcile conflicting goals among workers.
The Law of Fluency: Well-adapted cognitive work occurs with a facility that belies the difficulty of resolving demands and balancing dilemmas. The adaptation process hides the factors and constraints that are being adapted to or around.   Uncovering the constraints that fluent performance solves, and therefore seeing the limits of or threats to fluency, requires a contrast across perspectives.

Ned Hickling has challenged the universality of 'strong, silent automation is bad' i.e.  Mr Weasley's Law does not apply all the time. Disagreeing with Ned is fine. Just one problem. It means you are wrong. A proper response will appear, but after some thoughts on 'autonomy'.
The answer is likely to make use of Grote's thinking on zones of no control, whereby it is recognized that there are areas of automation where the operator has no effective control (cf. Ironies of Automation). For these zones, the operator is not held accountable, and accountability is assigned to the design authority, the operating organization or other agencies as appropriate.

"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." -- Mark Twain

Internet of Things - Use Cases (not)

Some use cases for consideration by the Internet of Things (IoT) community

Managerialism

Evolutionary managerialism - our current situation as a development of past bad habits

The wikipedia entry for managerialism is pretty good. It cites a definition by Robert R Locke.
"What occurs when a special group, called management, ensconces itself systemically in an organization and deprives owners and employees of their decision-making power (including the distribution of emolument), and justifies that takeover on the grounds of the managing group's education and exclusive possession of the codified bodies of knowledge and know-how necessary to the efficient running of the organization."
Locke's principal writing on the topic seems to be here, as 'Managerialism and the Demise of the Big Three' (pdf), and the book 'Confronting Managerialism'. The Big Three are the US automobile makers, and their demise is seen as being brought about by the Japanese management approach. Locke lays blame at the door of of neoclassical economics and  business school teaching.

This view of managerialism has strong precursors. Pfeffer and Sutton have pointed out the problems of US MBAs, what and how they teach. Mintzberg's  'Strategy Safari' has an account of the demise of the British motorcycle industry and Honda's success, including this quote from Hopwood:
"In the early 1960s the Chief Executive of a world famous group of management consultants tried hard to convince me that it is ideal that top level management executives should have as little knowledge as possible relative to the product. This great man really believed that this qualification enabled them to deal efficiently with all business matters in a detached and uninhibited way."
This ideal sounds like a job description of a UK generalist civil servant - still not dead 44 years after Fulton. Rory Stewart has written about managerialism in a number of public organizations e.g.  here and here.

A proper 21st Century dystopian view of managerialism as an entity in itself

Bruce Sterling has given a good description of a dystopian future as  'favela chic', a talk beautifully visualised here.  The connections between science fiction dystopia and collapsonomics are all too realistic for comfort.

The full horrors of managerialism as embodied in current global corporate capitalism have been captured in contemporary language by Rao, as 'The Gervais Principle', which starts with one of my favourite Hugh McLeod cartoons. The life cycle diagram seems to map well onto the more traditional life cycle at Adizes.

Rao's Guerilla Guide to Social Business is also available for download and is bang on the money. It includes a wonderful take on KM.

Managerialism and safety management

Managerialism appears to have penetrated safety management. One consequence is a concentration on hazards that are easily managed, at the expense of systemic hazards that require a resilient, learning, sensemaking approach. The sensemaking approach is set out in the 'The Learning School' in Mintzberg, or Weick and Sutcliffe (this link takes you to a great resource on High Reliability Organizations) and their book.

Managerialism in a safety context has been parodied all too accurately  by The Daily Mash here.

In a safety management context, managerialism looks deceptively innocent. The diagrams below look fine at first sight. Everything is organized. That is the problem. Being organized is vital, but it is not enough. Where, on these diagrams, can we find crew input, sensemaking, informal learning, trying things out? Not in 'the system'. This is the most difficult challenge facing the move to resilience.