The question as to why Mass incarceration starts around 1980 rather than 1866 is a good one. It’s a question that can be best answered IMO with a Marxist analysis.
Let’s start with the premise that the real power in America is not the politicians but the super rich and big business, or capital, actually shape the big picture decisions in a society. Let’s also suppose that capital is seeking the highest rate of return and it has no regard for pain and suffering among the masses. Let’s also assume that capital also wants to not be overthrown by the working class (which in this discussion includes not only all wage laborers of all races but also those who raise children and look after the home as well as unemployed people, basically anyone who is not an owner of capital). Lastly, let’s assume that while capital has a great deal of power to shape society to its advantage, it wants to optimize its position with as few resources expended as possible so capital is inclined to use existing social forces to further its goals.
So it’s America, it’s the mid 1970’s, you are a capitalist. You have enjoyed spectacular levels of profit by investing in or owning manufacturing companies in the US. You’ve enjoyed those returns since the mid 1940’s because after World War Two the US owned most of the world's industrial capital stock. You make enough profits to mollify the mostly white trade unions and still have plenty of money. You were able to pay high tax rates to pay for public services and still bring home plenty of money.
Near the end of this period, by the early 1970's your profits are declining. There is not enough money to go around any more so you must make it so you no longer have to pay for public services and you must break unions and start off shoring industry and converting the US economy to one based on finance and services. This way to can keep bringing home a lot of money even with profit margins falling. You use fear of black people to convince whites to give up unionization and to support divestment of public services. If black people can be in your union or can use your public swimming pools and attend your public school now, convince enough whites to get ride of those things.
That will create many problems. It will cause many households to be thrown into chaos, it will cause civil services to wither away and large swaths of the city as places with few jobs and almost no home ownership.
Capitalists found a few answers to manage the human cost an the political ramification that could follow so much dislocation for so many workers. A few of those solutions were carrots but most were sticks.
You coddle the professional-managerial class. You create new "BS" white collar jobs like those in marketing, PR and management consulting. You manipulate home prices so that those managerial workers can feel even more wealthy than before. However, capital mostly managed people with sticks.
The obvious means to do that was to massively increase police budgets, start a new drug war and start rapidly expanding prisons. This way you can cordon off the increasingly unemployed black population that lived in major cities. They or their parents or grandparents came to those cities during the great migration or during world war to to find employment in industry. With that industry gone, you know that those residents will largely be unemployed, under employed and those who are employed will have low wage and precarious employment in the service industry. The ruling class in the late 1970's and the 1980's had fresh memories of the black panthers, urban unrest in the 60's and the knowledge that many black men where Vietnam veterans. The war on drugs was revived in the 80's to have a pretext t demonize black people, crack become the official reason for increasing black poverty and unemployment. This gave a pretext for brutalization and surveillance of entire neighborhoods which prevented major insurrection and the specter of crack and crack addiction prevented solidarity with unemployed and precariously employed blacks and whites who were similarly dislocated by deindustrialization.
Mass incarceration becomes an additional means of control and intimidation. Prisons allow you to remove "surplus population," further the criminalization narrative for white audiences. It also has the benefit of creating jobs in largely white communities not in or near major cities. When places like upstate New York or the Central Valley of California or Downstate Illinois lost the industrial base in those areas, there was no way to create enough new jobs in finance, PR, marketing, management consulting so, jobs as prison guards becomes the substitute to avoid a situation where rural whites would fall into desperate poverty, migrate to cities and possibly become a left constituency. It's also worth noting that prisoners cannot vote in most States. They are counted as residents of where they are incarcerated in the census. So as the prison industrial complex is exploding, there is a political sorting happening where cities are becoming Democratic and rural ares are becoming Republican. By incarcerating black people in the inner city, you effectively take political power away from Democrats and give it to Republicans.
By the end of the 1980's, the ruling class is addicted to incarceration. In the last 30 years, mass incarceration has been a way to "solve" dislocation of workers and peasants from the global south. American foreign policy and neoliberalism makes life bad for people in Latin America and the US can lock them up if they try to come to the US looking for a better life. It has the benefit of creating another othered, underclass for white Americans to fear.
Neoliberalism also is always looking to privatize something and use it as a new revenue stream. Prisons services have become more and more privatized and they are a means to extract the little bit of extra money that the relatives of incarcerated people have. neoliberalism seeks low wages and prison labor is low and no wage so not only do capitalists get to have labor done for their companies and barely pay incarcerated people who do that work, they can use the threaten "off shoring" to America prisons to suppress wages of non incarcerated people. Neoliberalism also divests from services that help the masses but it still needs fire fighters to protect the property of rich people, rich people oftentimes have property on the edge of cities and climate change means increased fire risks in many areas. Mas incarceration provides a pool of slave labor to do the dangerous and dirty job of digging anti fire trenches around rich people's property whenever there are wildfires.
Mass incarceration may seem like a problem to us but for capitalists, who need to suppress wages, depress interracial worker solidarity, mollify the white proletariat, find new revenue streams and cultivate sources of very cheap and disposable slave labor, the warehousing of mostly black and brown bodies is a solution.
BTW apologies to anyone reading this who found parts of it as didactic. Much of what I said is pretty well known, I mostly wanted to incorporate elements into a whole and I wanted anyone reading this who is not super familiar with this topic to have some context.