Here’s how a Twitter engineer says it will break in the coming weeks

Here’s how a Twitter engineer says it will break in the coming weeks thumbnail

On November 4, just hours after Elon Musk fired half of the 7,500 employees previously working at Twitter, some people began to see small signs that something was wrong with everyone’s favorite hellsite. They saw it through retweets.

Twitter introduced retweets in 2009, turning an organic thing people were already doing–pasting someone else’s username and tweet, preceded by the letters RT–into a software function. In the years since, the retweet and its distant cousin the quote tweet (which launched in April 2015) have become two of the most common mechanics on Twitter.

But on Friday, a few users who pressed the retweet button saw the years roll back to 2009. Manual retweets were, as they were known, were returned .

The return of the manual tweet wasn’t Musk’s latest attempt at appeasing users. It was actually the first public crack in Twitter’s code base, a blip on the seismometer warning of a larger earthquake.

Twitter, a huge tech platform, is built on many interdependent components. “The more catastrophic failures are more entertaining, but the greatest risk is the smaller things starting degrade,” says Ben Krueger. A site reliability engineer with more than 20 years of experience in the tech industry, Krueger said. “These are very big, very complicated systems.” Krueger says one 2017 presentation from Twitter staff includes a statistic suggesting that more than half the back-end infrastructure was dedicated to storing data.

While Musk’s critics may hope that the platform will collapse like a thermonuclear weapon, it is actually gradual. If you’re familiar with the phenomenon, gradual crashes are a warning sign that a bigger crash is possible. That’s exactly what is happening right now.

It is the little things

Whether it’s manual replies appearing for a brief moment before retweets appear in their normal form, follower counts that race ahead the number of people actually following your tweets, or replies that refuse to load tiny bugs are appearing at Twitter’s periphery. Even Twitter’s rules ,, which Musk linked to on November 7, , was temporarily taken offline by millions of eyes. It’s becoming less reliable.

“Sometimes you’ll get notifications that are a little off,” says one engineer currently working at Twitter, who’s concerned about the way the platform is reacting after vast swathes of his colleagues who were previously employed to keep the site running smoothly were fired. This is why the engineer was granted anonymity to tell this story. After struggling with downtime during its “Fail Whale” days, Twitter eventually became lauded for its team of site reliability engineers, or SREs. However, this team was decimated after Musk’s takeover. The engineer says that while it is small, they add up to stability perceptions.

The small signs of something wrong will grow and multiply over time, he predicts. This is partly because the skeleton staff that remains to deal with these issues will soon burn out. He says that round-the-clock work is detrimental to quality and that he can already see it.

Twitter’s remaining developers have been tasked with maintaining the site’s stability over the past few days since the new CEO decided not to retain a large portion of its code base. As the company attempts to return to normalcy, they will spend more time addressing Musk’s (often taxing), whims for new features and products, rather than maintaining what’s there.

This is particularly problematic, says Krueger, for a site like Twitter, which can have unforeseen spikes in user traffic and interest. Krueger compares Twitter to online retail sites that allow companies to plan for large traffic events such as Black Friday with some predictability. He says that Twitter has the potential to have a Black Friday at any hour of the day. “At any given day, some news event can happen that can have significant impact on the conversation.” Responding to that is harder to do when you lay off up to 80% of your SREs–a figure Krueger says has been bandied about within the industry but which MIT Technology Review has been unable to confirm. The Twitter engineer agreed that the percentage sounded “plausible.”

That engineer doesn’t see a route out of the issue–other than reversing the layoffs (which the company has reportedly already attempted to roll back somewhat). He says, “If we’re going at a breakneck speed, then things will fall.” There is no way around it. The rate at which technical debt is accumulating is much faster than ever before. It’s almost as fast as financial debt.

He presents a dystopian future in which issues pile up and the backlog of maintenance tasks and fixes gets longer and longer. “Things are going to break. Things will break more often. Things will break for longer periods of times. He says that things will break down in more severe ways. “Everything will continue to grow until it’s unusable .” Twitter’s demise into an unusable wreck may be a while off, but there are already signs of process rot. It all starts with the little things: “Bugs in whatever client they’re using; any service in the backend they’re trying. These will be minor annoyances at first, but as back-end fixes are delayed, they will become more significant until people give up .”.

Krueger claims that Twitter will not go out of existence, but we’ll see more tweets not loading and accounts popping up and disappearing at will. He says that he would expect any data-processing system to have slowness, timeouts and other subtle failure conditions. They are often more subtle. They are also more difficult to find and fix.

The juddering manual replies and faltering followers are signs that this is already happening. Twitter engineers have created fail-safes to ensure that functionality doesn’t go completely offline, but that cut-down versions of it are available. Krueger says that this is what we are seeing.

Alongside minor glitches, Krueger believes there will be significant outages in the future. This is due in part to Musk’s drive for a reduction in Twitter’s cloud computing server load to try to recover up to $3 million per day in infrastructure costs. Reuters reports this project is called the “Deep Cuts Plan” and that it was created by Musk to reduce Twitter’s cloud computing server load in an attempt to recover up to $3 million per day in infrastructure costs. However, the engineer believes that there will be significant outages on the horizon. “A lot of the people I saw who were leaving after Friday have been there nine, 10, 11 years, which is just ridiculous for a tech company,” says the Twitter engineer. The decades of knowledge that Twitter’s systems work disappeared with these individuals as they left the offices. (Those within Twitter, and those watching from the sidelines, have previously argued that Twitter’s knowledge base is overly concentrated in the minds of a handful of programmers, some of whom have been fired. )

Unfortunately, teams stripped back to their bare bones (according to those remaining at Twitter) include the tech writers’ team. The engineer says, “We had good documentation thanks to [that team]”. It’s not so. It will be more difficult to determine the cause of problems when they occur.

Externally, it will be more difficult to find the answers. The communications team has been cut down from between 80 and 100 to just two people, according to one former team member who MIT Technology Review spoke to. The engineer says that there is too much work for them and they don’t know enough languages to deal effectively with the media. When MIT Technology Review reached to Twitter to discuss this story, they did not respond.

Musk’s recent criticism of Mastodon, the open-source alternative to Twitter that has piled on users in the days since the entrepreneur took control of the platform, invites the suggestion that those in glass houses shouldn’t throw stones. The Twitter CEO tweeted, then quickly deleted, a post telling users, “If you don’t like Twitter anymore, there is awesome site [sic] called Masterbatedone [sic].” Accompanying the words was a physical picture of his laptop screen open on Paul Krugman’s Mastodon profile, showing the economics columnist trying multiple times to post. Despite Musk’s attempt to highlight Mastodon’s unreliability, its success has been remarkable: nearly half a million people have signed up since Musk took over Twitter.

It’s happening at a time when the first cracks in Twitter’s edifice have begun to show. Krueger believes that this is just the beginning. He says, “I would expect to see significant public-facing issues with the technology within six month.” “And I feel like that’s a generous estimate.”

Read More