This is to say that we’re facing a coordination problem around executives being honest around the AI gains they’ve witnessed – if they co-operate, they keep their jobs. If they defect, they will possibly be fired by their embarrassed peers (who have now been implicitly called liars, cowards, or incompetents) and then replaced with someone that will toe the line anyway. If they could all admit the truth at once there might be some hope, but there is no way to coordinate that event.

I think it’s interesting to notice that this is not AI-specific. He kinda alludes to that in a different context but I think it’s important for people in the industry (and any industry really) to understand. This is driven by the incentives built into the system. Specifically the competition for profit. Whenever there’s a thing (technology, process, method) that promises significant profit, everyone and their mother jumps on it. For the simple reason that if they don’t, they’re getting replaced by someone who would. For the simple reason that if they don’t jump in, capital is going to leave their company and enter another one that promises to jump in, and therefore has higher expected profit. This can look as simple as large investors dumping your shares for the shares of a different company, sending your stock price in steep decline. Everyone competes for capital which requires competition for growing profits. Capital goes where the expected profits are higher. Without capital, firms can’t hire people and buy what they need to conduct business and create new products. This incentive makes it very difficult to avoid a situation like the one above. We’ve seen this in corporations with the move to the cloud, with outsourcing development to CHWTIA and so on.

On several occasions, we’ve been exposed to folks that have been sort of lukewarm on our main offerings, but they really, really wanted to use AI to perform a natural language query on their data. And we thought “Okay, if you really want to see it, maybe we can caveat this appropriately and show you what it might look like.”

This was a terrible mistake. It backfired in the most predictable way imaginable – every lukewarm client that saw the chatbot in action, even with us telling them that it was not going to accomplish what they wanted, wanted to buy it immediately. Every other consideration, including millions of dollars that we could plausibly help them achieve by non-AI means, was swept aside. It was like a dark and terrible force seized control of their limbs, plunged their hands into their own chests, and presented their still-beating credit cards to us in grim supplication.

The writing here (and everywhere) is phenomenal, and this section echoes something I’ve seen Cal Newport say before, something along the lines that somebody with no technical knowledge sees one neat trick and suddenly assumes AI will turn into a superintelligence deity.

I can’t name any specific people because they include personal friends

From the quote at the top. I keep seeing permutations of this and all I can say is these people are cowards