158 stories

Sam Bankman-Fried’s Truly Effective Philanthropy: Teaching

1 Comment

We should all recognize that Sam Bankman-Fried is much smarter than the rest of us. After all, outwardly he looks to be one of the biggest frauds of all time. By the age of 30 he amassed a fortune that dwarfs that of your average billionaire. He did it by running a crypto Ponzi-scheme. While claiming to be using his wealth to support philanthropies that were carefully selected to maximize human welfare, he was actually living a high life-style with his friends.

Now that the Ponzi has collapsed, the investors who trusted him look to be out of luck. And, of course there is no money for the philanthropies that he supported, many of which will are now struggling because they won’t get contributions they had been counting on.

That all looks pretty reprehensible, but maybe that’s the point. See, Sam Bankman-Fried was so committed to his philosophy of effective philanthropy that he was prepared to make himself appear to be the epitome of a despicable human being, and spend many years in prison, all to teach us that finance is a wasteful cesspool that needs to be reined in for the good of humanity. And, the place to start is his particular corner of the cesspool: crypto.

Philanthropy verse Reform: How Best to Save Humanity

The point here is straightforward. Suppose that Mr. Bankman-Fried was actually able to accumulate tens of billions of dollars through his brilliance, which he would then donate to the worthy causes he had carefully selected to have a maximum impact on human well-being. That would undoubtedly benefit some number of people in the United States and around the world.

But think for a minute about the financial sector. It has expanded enormously relative to the size of the economy over the last half century.

The broad finance, insurance, and real estate sector has more than doubled as a share of GDP over the last half-century, increasing from 5.5 percent of GDP in 1971 to 12.0 percent in 2021.[1] The additional 6.5 percent of GDP being devoted to finance in 2021 is equivalent to more than $1.4 trillion being absorbed by the sector. That comes to more than $11,800 a year for an average family.

The more narrow securities and commodity trading sector, along with investment funds and trusts, more than quadrupled as a share of GDP, rising from 0.55 percent of GDP in 1971 to 2.56 percent in 2021. This increase of 2.0 percentage points of GDP comes to more than $500 billion a year in the current economy, or almost $4,400 a year per family.

There is little to show for the massive expansion in the size of the financial sector. Finance is an intermediate good, like trucking. While both sectors are essential to the functioning of a modern economy, they don’t directly provide value to people in the way that the housing, food, or health care sectors do.

We need these sectors, but we want them to perform their economic functions as efficiently as possible. In the case of finance, those functions are facilitating payments to households and businesses and allocating capital to its best uses.

Clearly we have developed better mechanisms for paying our bills and carrying on other transactions, but the biggest developments are hardly new. Direct deposit of our paychecks and automatic payments for bills are great innovations that save lots of time for both sides of the transactions. However, these innovations date back more than four decades.

The same holds with credit cards and debit cards. The overwhelming majority of transactions are now made with these cards, but this is not especially new technology. Credit cards were already widely available in 1971, even if they were nowhere near as ubiquitous as they are today.   

We can give the financial sector credit for the increase in the convenience of our system of payments, but how much is this worth? Is the time saved from using credit cards or having a direct deposit of your payments worth $11,800 a year to you? That seems a bit steep. I suspect given the option, most people would prefer an extra $11,800 in their paycheck and be given the check by hand rather than having it deposited automatically in their bank account.   

How about the other part of the financial sector’s function, allocating capital to its best uses? There is no simple way to evaluate how effective our enlarged financial sector has been in allocating capital, primarily because we don’t have a counterfactual. We can’t point to an America with a smaller financial sector over the last half-century. (Steven Cecchetti and Enisse Kharroubbi did a cross-country analysis which found a larger financial sector boosted growth, but after reaching a certain size relative to the economy, it was a drag on growth.)

We can make a comparison of productivity growth in recent decades with productivity growth in the decades before the financial sector was consuming such a large share of the country’s output. In the years from the beginning of the Bureau of Labor Statistics productivity series in 1947 to 1972, productivity growth averaged 2.8 percent annually. From 1972 to 2022, productivity growth averaged just 1.8 percent.

If anything, productivity growth has slowed further as the financial sector has expanded relative to the economy. While there was a strong decade of productivity growth from 1995 to 2005, in the years from 2005 to 2019, productivity growth averaged just 1.4 percent.

The expanded financial sector may not be responsible for the slowing of productivity growth, and it’s certainly possible that it would have slowed even more without a larger financial sector. But, it is not easy to make the case that the financial sector has somehow led to faster productivity growth.

We pay for the waste in the financial sector not only through fees on financial transactions and our 401(k)s, and being front-run on our stock trades, but also through higher prices for housing and other items. Finance has created many of the great fortunes in the economy, not just Sam Bankman-Fried’s. When these people spend money buying bigger and/or more houses, it makes housing more expensive for the rest of us. They also hire people as their servants and demand workers for a wide range of activities from driving their cars to massaging their backs. Because the rich tie up so many workers meeting their luxury consumption, we have fewer people to work in child care centers or as teachers.

Sam Bankman-Fried Exposes the Corruption in Finance

So Bankman-Fried, being a genius, recognized the incredible waste and corruption in the financial sector. He knew that the best way to help humanity is to downsize the financial sector. The gains would dwarf the impact of anything he could hope to do with the money he could accumulate in his business dealings.

After all, even if he gave his entire Ponzi fortune of $15 billion to the best causes, this would be dwarfed by the good he could do by seriously downsizing the financial sector. After all, $15 billion is just over 1.0 percent of the $1.4 trillion increase in the relative size in the financial sector over the last half century. It is only 3.0 percent of the size of the increase in the relative size of just the narrow securities and commodities trading sectors.

Even if he managed to accumulate a pre-Twitter Elon Musk size fortune of $200 billion it would barely change the picture. That is less than 15 percent of the size of the bloat in the larger financial sector and just 40 percent of the bloat in the more narrow securities and commodities sector.

And, these are annual figures. The financial sector bloat is pulling these sums from the economy every year. A Musk-size fortune, accumulated over a life-time, would just be equal to 15 percent of a single year’s waste in the financial sector.

Obviously, Bankman-Fried is aware of this situation. He therefore realizes that the gains to humanity from reducing the waste in the financial sector would dwarf any benefits that he could hope to provide from the wealth he accumulates.

Bankman-Fried’s Brilliant Strategy

Recognizing the enormous waste and corruption in the financial sector, Bankman-Fried decided that the best way to attack it was by putting himself at the center of a scandal hitting finance at its most vulnerable point: the crypto craze. Most sectors of finance involve a mix of productive uses with speculation and waste. This is true of the stock and commodities markets, which do allow for businesses to raise capital and for primary goods producers to lock in prices, even if most trading is speculative in nature. Even private equity firms can occasionally turn around troubled businesses, as their supporters claim.

However, crypto does nothing for the economy. If all crypto currencies disappeared tomorrow, the only effect would be that some illicit transactions may become more risky for the people carrying them through.

This means that cracking down on crypto poses no real risks to the economy, only to crypto speculators. If we put a hefty tax on crypto trades, treating it like the gambling it is, it can raise revenue for the government and hugely reduce the amount of resources wasted in crypto trading.

Even more importantly, it can be a great foot in the door for a more general crackdown on finance. A crypto trading tax should introduce people in policy positions to the idea of taxing financial transactions (they should already be familiar with financial transactions taxes, but they aren’t), and ideally open the door to cracking down on finance more generally.

The potential benefits here are enormous. If we can just downsize the financial sector by 10 percent, it will free up more than $300 billion a year for productive purposes. That comes to more than $2,500 a year for every family in the country. As the effective philanthropy folks say, you can buy a lot of mosquito netting with $300 billion a year.

So, Bankman-Fried knew what he was doing in running a Ponzi-scheme and making himself look like one of the most despicable people alive. He may spend a lot of time in prison and be viewed with universal contempt for the rest of his life, but if his crimes lead to a crackdown on finance, he will have provided a great service to humanity.  

[1] These data are taken from National Income and Product Accounts Table 6.2B, with the total share being Line 52 divided by Line 1 for 1971 and Table 6.2D, Lines 57 and 62, divided Line 1 for 2021. For the narrow securities and commodity trading sector, and holding and trust accounts, the calculation uses Line 55 and Line 59, divided by Line 1 for 1971. For 2021, it uses Line 59 and Line 61, divided by Line 1. These tables only give data on labor compensation. The implicit assumption is that the industry’s value added is proportional to labor compensation in the sector. While this will not be precisely accurate, it should be reasonably close.  

The post Sam Bankman-Fried’s Truly Effective Philanthropy: Teaching appeared first on Center for Economic and Policy Research.

Read the whole story
16 hours ago
While a sarcastic parody at surface level, this is actually a great critique of #financialization - it's US centric but those familiar with the UK will know how much the overemphasis on London as a financial center really distorts that country's economy
Share this story

The Blithering Idiot

1 Share
Read the whole story
1 day ago
Share this story

Fermenting a revolution

1 Share

I believe this is the most important environmental technology ever developed. It might be all that now stands between us and Earth systems collapse.

By George Monbiot, published in the Guardian 24th November 2022

So what do we do now? After 27 summits and no effective action, it seems that the real purpose was to keep us talking. If governments were serious about preventing climate breakdown, there would have been no Cops 2-27. The major issues would have been resolved at Cop1, as the ozone depletion crisis was at a single summit in Montreal.

Nothing can now be achieved without mass protest, whose aim, like that of protest movements before us, is to reach the critical mass that triggers a social tipping point. But, as every protester knows, this is only part of the challenge. We also need to translate our demands into action, which requires political, economic, cultural and technological change. All are necessary, none are sufficient. Only together can they amount to the change we need to see.

Let’s focus for a moment on technology. Specifically, what might be the most important environmental technology ever developed: precision fermentation.

Precision fermentation is a refined form of brewing, a means of multiplying microbes to create specific products. It has been used for many years to produce drugs and food additives. But now, in several labs and a few factories, scientists are developing what could be a new generation of staple foods.

The developments I find most interesting use no agricultural feedstocks. The microbes they breed feed on hydrogen or methanol – which can be made with renewable electricity – combined with water, carbon dioxide and a very small amount of fertiliser. They produce a flour that contains roughly 60% protein, a much higher concentration than any major crop can achieve (soy beans contain 37%, chick peas, 20%). When they are bred to produce specific proteins and fats, they can create much better replacements than plant products for meat, fish, milk and eggs. And they have the potential to do two astonishing things.

The first is to shrink to a remarkable degree the footprint of food production. One paper estimates that precision fermentation using methanol needs 1,700 times less land than the most efficient agricultural means of producing protein: soy grown in the US. This suggests it might use, respectively, 138,000 and 157,000 times less land than the least efficient means: beef and lamb production. Depending on the electricity source and recycling rates, it can also enable radical reductions in water use and greenhouse gas emissions. Because the process is contained, it avoids the spillover of waste and chemicals into the wider world caused by farming.

If livestock production is replaced by this technology, it creates what could be the last major opportunity to prevent Earth systems collapse, namely ecological restoration on a massive scale. By rewilding the vast tracts now occupied by livestock (by far the greatest of all human land uses) or by the crops used to feed them – as well as the seas being trawled or gill-netted to destruction – and restoring forests, wetlands, savannahs, wild grasslands, mangroves, reefs and sea floors, we could both stop the sixth great extinction and draw down much of the carbon we have released into the atmosphere.

The second astonishing possibility is breaking the extreme dependency of many nations on food shipped from distant places. Nations in the Middle East, north Africa, the Horn of Africa and Central America do not possess sufficient fertile land or water to grow enough food of their own. In other places, especially parts of sub-Saharan Africa, a combination of soil degradation, population growth and dietary change cancels out any gains in yield. But all the nations most vulnerable to food insecurity are rich in something else: sunlight. This is the feedstock required to sustain food production based on hydrogen and methanol.

Precision fermentation is at the top of its price curve, and has great potential for steep reductions. Farming multicellular organisms (plants and animals) is at the bottom of its price curve: it has pushed these creatures to their limits, and sometimes beyond. If production is distributed (which I believe is essential), every town could have an autonomous microbial brewery, making cheap protein-rich foods tailored to local markets. This technology could, in many nations, deliver food security more effectively than farming can.

There are four main objections. The first is “Yuck, bacteria!” Well, tough, you eat them with every meal. In fact, we deliberately introduce live ones into some of our foods, such as cheese and yoghurt. As for disgusting, take a look at the intensive animal factories that produce most of the meat and eggs we eat and the slaughterhouses that serve them, both of which the new technology could make redundant.

The second objection is that these flours could be used to make ultra-processed foods. Yes, like wheat flour, they could. But they can also be used radically to reduce the processing involved in making substitutes for animal products, especially if the microbes are gene-edited to produce specific proteins.

This brings us to the third objection. There are major problems with certain genetically modified crops such as Roundup Ready maize, whose main purpose was to enlarge the market for a proprietary herbicide, and the dominance of the company that produced it. But GM microbes have been used uncontroversially in precision fermentation since the 1970s to produce insulin, the rennet substitute chymosin and vitamins. There is a real and terrifying genetic contamination crisis in the food industry, but it arises from business as usual: the spread of antibiotic resistance genes from livestock slurry tanks, into the soil and thence into the food chain and the living world. GM microbes paradoxically offer our best hope of stopping genetic contamination.

The fourth objection has more weight: the potential for these new technologies to be captured by a few corporations. The risk is real and we should engage with it now, demanding a new food economy that’s radically different from the existing one, in which extreme consolidation has already taken place. But this is not an argument against the technology itself, any more than the dangerous concentration in the global grain trade (90% of it in the hands of four corporations) is an argument against trading grain, without which billions would starve.

The real sticking point, I believe, is neophobia. I know people who won’t own a microwave oven, as they believe it will damage their health (it doesn’t), but who do own a woodburning stove, which does. We defend the old and revile the new. Much of the time, it should be the other way around.

I’ve given my support to a new campaign, called Reboot Food, to make the case for the new technologies that could help pull us out of our disastrous spiral. We hope to ferment a revolution.


Read the whole story
1 day ago
Share this story

How About Never?

1 Share

Powerful governments have no intention of preventing climate breakdown.

By George Monbiot, published in the Guardian 18th November 2022

The chances of any one person being born were calculated by the life coach Dr Ali Binazir. He multiplied the probability of your parents meeting, mating and conceiving by the chances of a particular sperm and egg fusing; of all your human and hominid ancestors reaching reproductive age; and of all them successfully reproducing. He arrived at a figure of one in 10 to the power of 2,640,000. In other words, a 10 followed by 2.6m zeros. It’s an unimaginable, miraculous number. Yet here we are.

The chances of being alive right now, as a member of one of the first generations to know the path it is on, and one of the last that can change it, must add several more zeros to this crazy number. The chances of being the president or prime minister of your nation at this critical moment … well you get the idea.

So how have heads of government chosen to use this miracle? To extend our time on Earth, earning the gratitude of all the improbable humans of the future? No. They have chosen to do nothing. Nothing that has a realistic chance, in this contest of probabilities, of changing our trajectory. They had a choice at the Cop27 meeting in Sharm el-Sheikh of defending the habitable planet or appeasing their sponsors. They went with the sponsors.

We know how way leads on to way, how the power amassed through corrupt decisions in previous generations drives the corrupt decisions of our time. We know that the licence granted to fossil fuel companies by 50 years of failure has enabled them to make stupendous profits – $2.8bn a day on average across that entire period – and that they need invest only a fraction of this money in politics to buy every politician and every political decision they want.

We know that the easiest way for a politician to secure power is to appease those who already possess it, those whose power transcends elections: the oil barons, the media barons, the corporations and financial markets. We know that this power appoints the worst possible people at the worst possible time. We know how, as elderly billionaires seek to grab ever more of the life that slips from them, they create a death cult.

Fifty years, you ask? Yes, the first international summit that claimed to address the environmental crisis took place in 1972. A handful of powerful nations, including the UK and US, convened what their secret minutes called an “informal and confidential” body at that summit, whose purpose, the notes show, was to ensure poorer countries did not get what they wanted, and that no international standards would be agreed on pollution or environmental quality.

They learned an important lesson there. You make the threats to your sponsors go away by nodding and smiling, saying the right things in public, then blocking effective measures behind closed doors. When they arrived at Cop27 this year, they had no intention even of paying the money they had promised to poorer nations to help them adapt – if such a thing is possible – to climate breakdown, let alone seeking to prevent that breakdown from happening.

So here we are, after 50 years of engineered failure, with not one of the 40 markers of climate action on track to meet the targets governments have agreed. In the first nine months of this year, the seven biggest private sector oil companies made around $150bn in profits. Yet governments continue to supplement this loot by granting oil and gas companies $64bn a year in public subsidies.

There are no longer any feasible means of preventing more than 1.5C of global heating if new oil and gas fields are developed. Yet fossil-fuel companies, with the encouragement of the governments that either own or license them, are planning a major investment surge between 2023 and 2025. The biggest planned expansions, by a long way, are in the US. The soft facts – the vague and unsecured promises at Sharm el-Sheikh about curbing consumption – count for nothing against the hard facts of extending production.

We no longer need to speculate about where this path might lead: we have stepped through the gates. The floods in Pakistan that displaced 33 million people and washed away 3 million acres of soil followed a crop-shrivelling heatwave. This is the whipsaw effect predicted in scientific papers: of moderate weather giving way to a violent cycle of extremes. It’s hard to see how the country will ever recover from the economic shocks of these disasters: as it starts to pick itself up, it’s likely to be knocked down by another one. China this year, though this was sparsely reported in the western media, suffered not only the greatest heatwave in its instrumental record, but the greatest heat anomaly ever recorded anywhere. The devastating drought in the Horn of Africa, now in its fifth year, offers a glimpse of what “uninhabitable” may look like.

The rich world’s governments arrived at the conference in Egypt saying “it’s now or never”. They left saying “how about never?”. We sail through every target and objective, red line and promised restraint towards a future in which the possibility of anyone’s existence starts to dwindle towards zero. Every life is a madly improbable gift. For how much longer will we sit and watch while our governments throw it all away?


Read the whole story
4 days ago
Share this story

Cliff Edge

1 Share

Here’s how it happened before – a mass extinction caused by fossil fuel burning. It will happen again unless we step back from the precipice.

By George Monbiot, published in the Guardian 3rd November 2022

Budleigh Salterton, on the south coast of Devon, sits above the most frightening cliffs on Earth. They are not particularly high. Though you don’t want to stand beneath them, they are not especially prone to collapse. The horror takes another form. It is contained in the story they tell. For they capture the moment at which life on Earth almost came to an end.

The sediments preserved in these cliffs were laid down in the early Triassic period, just after the greatest mass extinction in the history of multicellular life, that brought the Permian period to an end 252m years ago. Around 90% of species died, and fish and four-footed animals were more or less exterminated between 30 degrees north of the equator and 40 degrees south.

Most remarkably, while biological abundance (if not diversity) tends to recover from mass extinctions within a few hundred thousand years, our planet remained in this near-lifeless state for the following 5m years. In studying these cliffs, you see the precipice on which we teeter.

The lowest stratum at the western end of the beach is a bed of rounded pebbles. These are the stones washed off Triassic mountains by flash floods and deposited in great dumps by temporary rivers. Because the forests and savannahs that might have covered the mountains had died, there was nothing to hold the soil and subsoil together, so erosion is likely to have accelerated greatly.

At the top of the pebble bed is a stony desert surface. The pebbles here have been sculpted by the wind into sharp angles and varnished with shiny oxides, suggesting the surface was unchanged for a long time. Above it are towering red Triassic sand dunes. Through a quirk of erosion, these soft deposits have been sculpted into hollows that look uncannily like fanged and screaming skulls.

We now know that there were two main pulses of extinction. The first, which began 252.1m years ago, mostly affected life on land. It coincided with a series of massive volcanic eruptions in the region now known as the Siberian Traps. The second, more devastating phase, started about 200,000 years later. It almost completed the extinction of terrestrial life, as well as wiping out the great majority of species in the sea.

Though we cannot yet be sure, the first phase might have been triggered by acid rain, ozone depletion and metal pollution caused by volcanic chemicals. As rainforests and other ecosystems were wiped out, more toxic compounds were released from exposed soils and rocks, creating an escalating cycle of collapse.

The second phase appears to have been driven by global heating. By 251.9m years ago, so much solidified rock had accumulated on the surface of the Siberian Traps that the lava could no longer escape. Instead, it was forced to spread underground, along horizontal fissures, into rocks that were rich in coal and other hydrocarbons. The heat from the magma (underground lava) cooked the hydrocarbons, releasing vast amounts of carbon dioxide and methane. In other words, though there were no humans on the planet, this disaster seems to have been caused by fossil fuel burning.

Temperatures are believed to have climbed by between 8C and 10C, though much of the second phase of extinction might have been caused by an initial rise of between 3C and 5C. The extra carbon dioxide also dissolved into the oceans, raising their acidity to the point at which many species could no longer survive. The temperature rise appears to have brought ocean currents to a halt, through the same mechanism that now threatens the Atlantic meridional overturning circulation, which drives the Gulf Stream. As wildfires raged across the planet, incinerating the vegetation protecting its surface, ash and soil would have poured into the sea, triggering eutrophication (an excess of nutrients). In combination with the high temperatures and stalled circulation, this starved the remaining life forms of oxygen.

A paper released as a pre-print in September might explain why recovery took so long. Because so many of the world’s rich ecosystems had been replaced by desert, plants struggled to re-establish themselves. Their total weight on Earth fell by about two thirds. Throughout these 5m years, no coal deposits formed, as there wasn’t sufficient plant production to make peat bogs. In other words, the natural processes that remove CO2 from the atmosphere and turn it into wood and soil or bury it as fossil carbon stalled. For 5m years, the world was trapped in this hothouse state. In the cliffs at the eastern end of the bay, you can see when conditions began, at last, to change, as the fossilised roots of semi-desert plants twist down through the ancient sand dunes.

The story the cliffs tell is of planetary tipping points: Earth systems pushed past their critical thresholds, beyond which they collapsed into a new equilibrium state, that could not be readily reversed. It was a world hostile to almost all large life forms: the monsters of the Permian were replaced nearly everywhere by dwarf fauna.

Could it happen again? Two parallel and contradictory processes are in play. At climate summits, governments produce feeble voluntary commitments to limit the production of greenhouse gases. At the same time, almost every state with significant fossil reserves – including the UK – intends to extract as much as they can. A report by Carbon Tracker shows that if all the world’s reserves of fossil fields were extracted, their combustion would exceed the carbon budget governments have agreed sevenfold. While less carbon is contained in these reserves than the amount produced during the Permian-Triassic extinction, the compressed timescale could render this release just as deadly to life on Earth. The increase in atmospheric CO2 at the end of the Permian took about 75,000 years, but many of our fossil fuel reserves could be consumed in decades. Already, we seem to be approaching a series of possible tipping points, some of which could trigger cascading collapse.

Everything now hangs on which process prevails: the sometimes well-meaning, but always feeble, attempts to limit the burning of fossil carbon, or the ruthless determination – often on the part of the same governments – to extract (and therefore burn) as much of it as possible, granting the profits of legacy industries precedence over life on Earth. At the climate summit this month in Egypt, a nation in which protests are banned and the interests of the people must at all times cede to the interests of power, we will see how close to the cliff edge the world’s governments intend to take us.


Read the whole story
5 days ago
Share this story

Why Twitter Didn’t Go Down: From a Real Twitter SRE


Twitter supposedly lost around 80% of its work force. What ever the real number is, there are whole teams with out engineers on it now. Yet, the website goes on and the tweets keep coming. This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I’d like to explain my little corner of Twitter (though it wasn’t so little) and some of the work that went on that kept this thing running.

Background and History

For five years I was a Site Reliability Engineer(SRE) at Twitter. For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it. (There might be only one or two other people)

A cache can be used to make things faster or to alleviate requests from something that is more expensive to run. If you have a server that takes 1 second to respond, but it’s the same response every time you can store that response in a cache server where the response can be served in milliseconds. Or, if you have a cluster of servers where serving 1000 requests a second might cost $1000, you can instead use the cache to store the responses and serve it from that cache server instead. Then you would have a small cluster for $100 and a cheap and large cache cluster of servers maybe for another $100. The numbers are just examples to illustrate the point.

The caches took on most of the traffic the site saw. Tweets, all of the timelines, direct messages, advertisements, authentication, all were served out the Cache team’s servers. If something went wrong with Cache, you as a user would know, the problems would be visible.

When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names. I am happy to say operations on that team is not like that anymore!

How the cache’s keep running

The first big point that is keeping the caches running is that they are ran as Aurora jobs on Mesos. Aurora finds servers for applications to run on, Mesos aggregates all the servers together so Aurora knows about them. Aurora will also keep applications running after they are started. If we say a cache cluster needs 100 servers, it will do its best to keep 100 running. If a server completely breaks for some reason, Mesos will detect this, remove the server from its aggregated pool, Aurora will now be informed that there are only 99 caches running and then know it needs to find a new server from Aurora to run on. It will automatically find one and bring the total back to 100. No person needs to get involved.

In a data center servers are put into things called racks. Servers on racks are connected to other servers on racks through a device called a switch. From here there is a whole complex system of connections of switches to more switches and routers and eventually out to the internet. A rack can hold somewhere between 20 to 30 servers on it. A rack can fail, the switch can break or maybe a power supply dies, this then takes down all 20 servers. One more nice thing Aurora and Mesos does for us is ensure that not too many applications will be put on a single rack. So the whole rack can go down safely and suddenly, Aurora and Mesos will find new servers to be homes for the applications that were running there.

That spreadsheet mentioned before, it was also tracking how many servers were on racks and the spreadsheet writer tried to make sure there weren’t too many. Now with the current tools, when we provision new servers to make them live, we have tools that will track all of this. Those tools make sure the team doesn’t have too many physical servers on a rack and that everything is distributed in a way that won’t cause problems if there are failures.

Mesos doesn’t detect every server failure unfortunately, so we have extra monitoring for hardware issue. We look for things like bad disks and faulty memory. Some of these won’t take down a whole server but might just run slow. We have a dashboard of alerts that gets scanned for broken servers. If one is detected to be broken, we automatically create a repair task for someone in the data center to go look at it.

One more piece of important software that the team has is a service that tracks up time of cache clusters. If too many servers have been seen as down in a short period of time, new tasks that require taking down a cache will be rejected until it is safe. This is how we avoid accidentally taking down entire cache clusters and overwhelming the services that are protected by them. We have stops in place for thing like too many to quickly being down, too many out for repair at one time or Aurora not being able to find new servers to place old jobs. To create a repair task for a server detected as broken, first we check if it is even safe to remove jobs from it by checking that service, then once it is empty it is marked safe for a data center technician to work on it. When the technician in the data center marks the server as fixed, we again had tools that looked for this and automatically activated the server so it could run jobs. The only human needed was the person in the data center actually fixing it. (Are they still there though?)

Repeated application issues were also fixed. We had bugs where new cache servers wouldn’t be added back(race condition on start up) or sometimes it took up to 10 minutes to add a server back(O(n^n) logic). Since we weren’t bogged down by manual tasks thanks to all of this automation work we could develop a culture in the team where we could go and fix these while keeping projects on track. We have other automatic fixes, for example, where if some application metrics like latency were an outlier we automatically restarted the task, so an engineer wouldn’t get paged. The team would maybe get one page a week, almost never critical. We frequently had on call rotations where no one got paged.

Capacity planning was also one of the more important reasons why the site hasn’t gone down. Twitter has two data centers running that can handle the entire site being failed into it. Every important service that runs can be run out of one data center. The total capacity available at anytime is actually 200%. This is only for disaster scenarios, most of the time both data centers are serving traffic. Data centers are at most 50% utilized. Even this would be busy in practice. When people calculate their capacity needs, they figure out what is needed for one data center serving all traffic, then normally add headroom on top of that! There is a ton of server headroom available for extra traffic as long nothing needs to be failed over. An entire data center failing is pretty rare, it only happened once in my five years there.

We also kept cache clusters separate. We didn’t have a multi tenant clusters that served everything and had application level isolation. This helps so if one cluster is having an issue it kept the blast radius to only that cluster and maybe some co-located servers on the machine. Again, Aurora helps here by keeping caches distributed so there won’t be a lot effected and eventually monitoring will catch up and fix them.

What would you say ya do here?

Well, I did everything above! I did talk to the customers (teams that used cache as a service). After things were automated, I automated more. I also worked on interesting performance issues, experimented with technology that might make things better and drove some large cost savings projects. I did capacity planning and determined how many servers to order. I was pretty busy. Despite what some think, I wasn’t just collecting paychecks while playing video games and smoking weed all day.

That’s how the caches that are serving Twitter requests are staying up and running. This is only a part of what day to day operations are like. It was a lot of work over the years to get to this point. This is a moment to step back and appreciate that this thing is still actually working!

Well for now at least, I’m sure there’s some bugs lurking somewhere...

Read the whole story
6 days ago
Share this story
Next Page of Stories