Categories
English

Renaissance of Intelligence

For people who are interested in artificial intelligence, the past decade feels like another Renaissance — the boundaries between humans and machines are repeatedly re-defined by the invention of new technologies, from machines that could beat world Champions in games to AI assistants that could talk like real humans.

This article shares some stories behind this incredible AI Renaissance. The sources of the stories include my first-hand observation in the field as well as Cade Metz’s recent book  Genius Makers, which I highly recommend.

A Sputnik Moment

A key reason for the current revolution is the re-invention of deep learning, a technology that simulates human brains as complex network architectures through computers.

The idea is not new. Scientists have been searching for the truth of human intelligence for a long time, and a natural starting point is our brain — the only intelligent machinery built by our mother nature. Artificial neural networks, predecessors of deep learning, were very popular between the 1950s and 1980s but lost their popularity because data was not enough and the computers then were too weak to solve any interesting problems.

It would take another two decades before its revival. In the late 2000s, a group of young scientists started to connect the power of the booming Internet with artificial intelligence research. In 2009, an assistance professor of Princeton Dr. Fei-fei Li1 compiled a vast database of Internet images (a.k.a. ImageNet dataset). The ImageNet dataset soon became the benchmark for Computer Vision, a subfield of artificial intelligence. In 2012, Geoffroy Hinton and his team significantly improved the metrics by more than ten percent, a jaw-dropping achievement that was a magnitude higher than any previous improvement.

This was a “Sputnik moment” for the artificial intelligence research community, at the time the mainstream research direction was for scientists to figure out the solution and to program the software based on the solution. Hinton’s success in the ImageNet challenge showed that an alternative approach — letting the neural networks learn a solution without prescription from humans — would work better. 

Godfather is Heading to Industry

Huge amounts of data and computer resources were essential to this success. No one knew this better than Geoffroy Hinton himself, who is also known as the Godfather of deep learning. Hinton was one of the early persons that popularized back-propagation, a fundamental algorithm used to train neural networks. When the field entered into a winter between the 1990s and early 2000s, most researchers switched to other research directions due to scarce funding sources. However, Hinton was still a stubborn proponent of the idea and was trying to revive it.

Hinton knew that his research would need resources from elsewhere, and only the big Internet companies had the pocket deep enough and data big enough to make the idea work. In addition to his academic achievement, Hinton also had great business savvy. With his two students, Hinton founded the DNNResearch company in 2012 and soon decided to sell it to big Internet companies. The book Genius Makers gave a vivid description of how Hinton orchestrated the auction in a Lake Tahoe hotel and how tech companies all over the world wooed him. DNNResearch was eventually acquired by Google for 44 million US dollars.

More important than the price tag is the precedence that Hinton created. To lure Hinton, Google allowed him to keep his position on both sides — but he had to be an “intern” in Google to work around the company’s rules. Before Hinton, it was rare for eminent researchers to work for tech companies because of the fear of losing their tenured positions in universities. Soon after the purchase, a lot of AI researchers followed Hinton’s example to join technology companies, including Yann LeCun,  another deep learning pioneer who later led Facebook’s AI lab, Andrew Ng, who led the research lab in Baidu. What’s more, following their advisors, students from various research labs flocked into big technology companies.

Among the technology companies, Google (and its parent company Alphabet) stood out for its unparalleled role in this wave of AI Renaissance. Its research divisions — Google Brain and DeepMind — are the driving force behind a lot of the greatest breakthroughs. What’s more, the fact that it could use AI to create so many profitable applications has a demonstration effect on all other companies.

One important person behind this is Jeff Dean, a legendary engineer that laid the foundation of Google’s infrastructure. In 2011, Andrew Ng introduced the deep learning concept to Jeff and he was intrigued immediately.  Jeff was looking for his next application and deep learning was a perfect one.  Andrew, Jeff, and another researcher Greg Corrado founded the Google Brain team. As a founding engineer of Google, Jeff has a great influence on Google’s management team and also enjoys enormous popularity within its engineering and research organizations (so much so that people made fun of him by creating the “Jeff Dean facts“). Jeff created an umbrella where the Google Brain team could operate without worrying about anything else.

In Google Brain, Andrew Ng and his colleagues helped create the system that could learn the “cat” concept from millions of YouTube videos, which drew a lot of media attention and publicized the field.  Andrew Ng eventually left the Brain team to work on his own startup. But he recommended Geoffroy Hinton as his replacement, which triggered the DNNResearch acquisition. Under the leadership of Jeff and Hinton, Google Brain significantly contributed to the field by both pushing the research frontiers and publishing the TensorFlow framework that makes the technology accessible to outside communities.

New World Champion

One limitation of the techniques Hinton was trying (a.k.a. supervised deep learning) was that it requires datasets labeled by humans. Demis Hassabis co-founded DeepMind to address this limitation and explore other applications. He wanted to build a system that doesn’t depend on human supervision and could perform better than humans. A child prodigy in Chess, Hassabis believes that games are the best starting points. Although games had been a proving ground for AI since the 50s, no one has been more committed and successful than Hassabis in this direction.

Hassabis and his DeepMind team combined deep learning with reinforcement learning, a technology that allows computers to adapt their behaviors through trial and error (the same way we humans learn). With this new technology, DeepMind built a system that could learn the nuances that were never found by humans before in popular video games like Breakout and published their results in Nature. This publication drew the attention of  Google’s management. In 2014, Google purchased DeepMind for more than $500M — this time both Hinton and Jeff are on the buyer side. With the resources from Google, DeepMind doubled down on its mission. In May 2017, DeepMind’s AlphaGo AI beat the world champion Ke Jie. Since then, it has kept beating humans in one field after another.  In addition to DeepMind, Google’s other AI division also released the BERT system that significantly improved performance in natural language tasks.

There were also a lot of breakthroughs outside Google and DeepMind. For example, OpenAI, which was co-founded by some of Hinton’s students and Silicon Valley elites like Elon Musk and YC CEO Sam Altman, tackled many other games and robotics applications through reinforcement learning and they released the language models that achieved amazing results. The successes of Google/DeepMind/OpenAI and other AI research teams have brought the public interest in AI to an unprecedented level.

What’s ahead?

A keen observer would find that the current AI Renaissance consists of many small cycles. Each cycle starts when a difficult yet well-defined benchmark problem is solved. Thanks to the huge public attention, the research team that solved the problem would be able to get a huge amount of resources to continue their research. The team then tackles the next more challenging benchmark problem with a larger model.  The cycles were started by academics and their students and were reinforced by big technology companies. People knew, either consciously or unconsciously, that it was the best way of attracting attention, funding, and talents.

Notwithstanding, ImageNet and Go games are still not real-world problems. In addition, there have been increasing concerns that this type of AI research pattern has caused enormous resource consumption and has made the AI models to be overly complex.For example, the GPT-3 language model related by OpenAI includes 175 billion parameters and each train takes around 4.6 million dollars.  In addition, many AIs that overfit man-made tasks turn out to perform poorly in many real-world applications.

We should and would break such cycles. Building cost-effective AI and making it really work in real-world applications is crucial to keep the movement going. In the next decade, there will be a lot more exciting stories ahead of us.

Disclaim: All opinions are mine and not endorsed by my current or previous employers.


  1. Fei-fei Li was an assistant professor of Princeton University at the time but moved to Stanford later. The original version called Fei-fei Li a Stanford professor by mistake, thanks Jike Chong for pointing it out.
Categories
English

In Search of Memory

Eric Kandel won the Nobel prize in 2000 for his contribution to the understanding of memory at the molecular level. His autobiography, In Search of Memory, describes both his experience of escaping from Austria to America and his inquiry into the science behind our memory throughout his career. This essay is created based on the autobiography and extensive researches on the web.

Escape from Holocaust

Eric was born in a Jewish family in Vienna. As a Jew, Eric’s childhood was dominated by the Nazi’s growing influence in Austria. From its very beginning of the 1920s, the Nazi party aimed to merge all German-speaking people into a Greater Germany. In 1937, Hilter forced the Austrian chancellor Schuschnigg to resign and sent his troops to occupy the country, which was the largest German-speaking state outside Germany. The event, known as Anschluss, was welcomed by the Austrian Germans because a lot of them felt Austria was not fairly treated in the Treaty of Saint-Germain signed after WWI. After the Nazis took the power in Austria, a lot of Austrian Jews were forced to leave the country due to the violence targeting them (e.g., Kristallnacht). Thanks to the help of a local Jewish organization Kultusgemeinde, Eric’s family was able to emigrate to the United States of America in 1939. The young Eric was only Nine by then.

Hitler announces the Anschluss on the Heldenplatz, Vienna, 15 March 1938.

Most of the Jews who didn’t escape Nazi Austria became the victims of the Holocaust. The experience of escaping the Holocaust greatly influenced Eric throughout his whole life. Later after Eric won the Nobel prize in 2000, he used his influence to press the Austrian government to recognize the misfortune of the Jew community during Anschluss, which was largely ignored post-WWII, and to advocate the rights of Jews community in the country.

America

After emigrating to the US, Eric finished his education first in a Jewish school and then attended Harvard College. There Eric was attracted to psychoanalysis because it was imaginative, comprehensive, and empirically grounded. His attraction to psychoanalysis was further enhanced by the fact that its founder Freud was Viennese and Jewish and had been forced to leave Vienna. He later enrolled in New York University and aspired to become a psychoanalyst.  In the fall of 1955, Eric decided to take an elective at Columbia University with the neurophysiologist Harry Grundfest. Since then, Eric’s research career gradually shifted to find the biological basis of mental function.

Eric is particularly interested in the formation of memory. In 1890, William James concluded that memory must have at least two different processes: a short-term process and a long-term process. The basic units of the brain are the neurons, which are connected through synapses. Signals of one neuron are passed to the next neuron through chemical neurotransmitters that are available in synapse. One common hypothesis is that short-term memory is stored as the distribution of neurotransmitters across different synapses. A stimulus would activate a spatial pattern of activity across neurons in a brain region,  which will deplete the neurotransmitters. The distribution of neurotransmitters will form a trace of the stimuli, which is the short-term memory.

The short-term memory trace decays over time as neurotransmitters are re-generated. As a result, short-term memories need to be consolidated to long-term storage. Behavioral experiments suggest it happens through repetition — what is well known as “Practice makes perfect”.

Scientists also realized the importance of the hippocampus in turning short-term memory into long-term memory thanks to the extensive research on Henry Molaison (H.M.), who is probably the most famous patient in the history of memory research. After a treatment operation in which his hippocampus was removed, H.M.’s intelligence was intact, yet he lost the ability to form new memories. Other than this vague picture, Scientists had very little knowledge of the exact biochemical process of memory. It was under such a background that Eric entered the domain of memory research.

Most of Molaison’s two hippocampi were removed bilaterally.

Aplysia

The first question Eric needs to answer is how neurons could adjust their connections based on environmental stimuli. Unfortunately, Human brains are too complex for any thorough analysis, each human brain has about 100 billion neurons. As a result, Eric experimented on Aplysia instead, whose brain has only about 20,000 cells, making it a perfect model animal to analyze how neurons work. In 1962,  Eric joined the lab of a French scientist Ladislav Tauc, one of the few scientists who worked on Aplysia then, as a post-doc to learn about this interesting sea slug. Eric’s work on Aplysia has laid the foundation for understanding the mechanism of memory — so much so that Eric presented a picture of Aplysia wearing a Nobel medal during his Nobel prize ceremony,

“Aplysia Won the Nobel Prize”

In Search of Memory

Eric and his team realized that long-term memories are formed as anatomical changes of the neurons. A single neuron has approximately 1300 presynaptic terminals (only 40% of which are active) with which it contacts about 25 different target cells. Through the consolidation process, the creates long-term memory, both the percentage of active presynaptic terminals and their total number. The number of synapses changes during learning. Memory is recalled when a certain sensory stimulus triggers the “reads out” of the new state of the synapse, which has been altered by learning.

In 1953, Waston and Crick proposed the famous Double Helix model of DNA, which opened the new world of molecular biology.  In the memory-research field, Louis Flexner from the University of Pennsylvania discovered that applying a drug that inhibits the synthesis of proteins would disrupt long-term memory. Eric realized that the same process also applies to Aplysia and that long-term memory storage requires the synthesis of new proteins.

One revolutionary breakthrough in molecular biology was the realization that gene function can be regulated up and down in response to environmental signals. Inspired by this breakthrough, Eric continued to investigate genes’ role in learning and memory formation.  Through researching Aplysia, Eric and his team realized that long-term memory is formed through switching on and off certain genes that increase or inhibit the growth of certain synapses.

For decades, Kandel has been studying how we create short-term and long-term memories at the molecular level. His work helps reveal the full picture of the memory-forming mechanism:

  1. The memory storage takes place in at least two stages: A short-term memory lasting minutes is converted — by a process of consolidation that requires the synthesis of new protein — into stable, long-term memory lasting days, weeks, or even longer. 
  2. A single stimulus strengthens the synapse through the depletion of neurotransmitters, which form the short-term memory.
  3. Repeated stimulation causes certain genes to be switched on and the growth of new synapses, which creates long-term memory.

Eric’s journey from a refugee from Austria to a Nobel Laureate is a great example of how the tolerant and open environment of America could release boundless energy from immigrants like Eric and inspire them to think in new ways. In contrast, the city of Vienna, once a center of art and science, lost its glory under the suppressive occupation by the Nazis. His experience is still important for us after a hundred years.

Categories
English

What modern business leaders could learn from Genghis Khan?

Genghis Khan as portrayed in a 14th-century Yuan era album;

Genghis Khan created the Mongol Empire, the biggest empire in human history. At its height, the Mongol Empire covered a land area of more than 9.15 million square miles and a population of more than 100 million. Another surprising fact is that the population of Mongols was only a few million.

Why were Mongols able to conquer the world with such a tiny population? One important reason is that Genghis Khan created a specialized organization that could leverage the most advanced technology at the time (Mongolian horses) to solve the most ambitious problem (conquering the world). 

Despite the small number of human soldiers, there were a huge number of horses in the Mongol army. Each Mongol soldier has 3-4 Mongolian horses at his disposal at any time. Mongolian horses had very great endurance and were the most advanced military technologies during the cold-weapon era. In contrast, their enemies either don’t have any horses or could only use inferior horses. 

More importantly, Genghis Khan organized his Mongol soldiers in a way that could leverage the advantages of those Mongolian horses to the full extent. The command structure of the Mongol army was much more flexible than other armies during the period. Lower-level leaders have significant license to execute orders in the way they considered best. The super flexible organization allowed Mogol armies to attack en masse, divide into smaller groups to encircle and lead enemies into an ambush, or divide into small groups to mop up a fleeing and broken army. Because they could fully leverage the mobility of horses, a few Mongolian cavalry soldiers could easily defeat hundreds of foot soldiers.

Thanks to horses, the Mongolian army could cover up to 100 miles (160 km) per day, which was unheard of by other armies of the time. Mongolian soldiers were able to travel thousands of miles without stopping by rotating horses during the trip. Because of such great mobility, the Mongol empire could allocate resources on a global scale to defeat every local enemy. For example, the Mongols were able to fight with both the Muslim world and China at the same time. After Mongols conquered Muslims, they were able to leverage the technology they got from Muslims (like the counterweight trebuchet) to destroy the Song dynasty.

Genghis Khan and his Mongolian armies have taught us two things:

  1. New technology requires a new form of human organization to fully leverages its power. 
  2. An organization that could leverage the new power would be able to unlock even more new opportunities.

In the past few decades, we are creating new technologies to extend our brains. One notable new technology is artificial intelligence (AI), which allows machines to make predictions and decisions autonomously.  The relationship between the new AI tools and humans is similar to horses and Mongolian soldiers.

A business would need to transform its organizational structure to fully leverage the power of AI tools.

  1. For a lot of traditional businesses, the bottom of the organizational chart is a huge number of employees who work on operational tasks. As a result, management is based on carrots and sticks. More advantage management like (motivation alignment) is only available for strategic positions.
  2. In AI-first organizations, even junior employees will have hundreds of AI tools at his/her disposal and their influence on the organization is equivalent to a much higher-level person in those traditional organizations. Organizational management needs to be more motivation-driven throughout the organization.  The organization also (is able to ) and needs to be leaner and flatter, which encourages innovation.
The hidden workforce of AI-first organizations

Proactively leveraging AI tools not only reduces cost but also unleashes new powers (like horses do to Genghis Khan’s troops).  

  1. The natural way of organizational growth is to throw hiring humans. However, more people would create a communication burden and operational overhead. As an organization grows, the Return-On-Investment (ROI) of extra hiring will eventually decrease to be below 1, which prevents the company to scale further. 
  2. “Hiring” AI systems, in contrast, would not incur extra overhead. What’s more, AI systems typically get smarter as more people use them. As a result, the ROI will increase as the usage of the AI system increases.

The only ceiling floor for the scaling of an AI system is from the technical side. Currently, most of the commercial-viable AI system is only designed for a single problem. And for most of the problems, AI systems haven’t reached the human-level yet. This will be a bottleneck in the foreseeable future but more and more AI systems will be invented as time goes by. Human + AI collaboration would be a strong disruptive power for industries in which AI solutions are available. 

Hiring more people doesn’t make the manager’s job redundant. Instead, it makes their jobs more important. Similarly, the adoption of AI systems doesn’t make their users redundant. They will increase the scope of their users and the whole organization. Humans are tremendously flexible and could always find creative new usage of new capabilities.  For example, AI may be able to help doctors to diagnose basic medical conditions, but it won’t be able to replace doctors. Instead, doctors would be able to focus on more complicated medical problems. As long as humans haven’t reached immortality, there are always new problems for doctors to solve.

We don’t want another Mongol empire that causes deaths, but we do need business growth that could make human life better.  In addition to scaling the human part of the organization, every business leader should also consider where their “horses” are and how to provide organizational support to enable employees to use them.

Reference

  1. Mongol military tactics and organization
  2. Wikipedia: Mongol military tactics and organization.
  3. “The Mongol Empire’s Best Weapon: The Mongolian Horse” History on the Net © 2000-2021, Salem Media.
Categories
English Newsletter

Conan’s Newsletter No. 16

Book of the Week: Shoe Dog

I typically recommend a book only if it is worth reading multiple times. Shoe Dog by Nike co-founder is one of such books. I finished reading this book a few months back when my manager recommended it and completed another pass when I was on a road trip to Death Valley. Every read gives me fresh thoughts because Phil is a great storyteller, and Shoe Dog is not a typical memoir.

The book starts with a vivid description of Phil’s world trip right before he founded Nike. In the 1960s, German products dominated the American sports shoe market. As a former track runner at college, Phil envisioned that Japanese running shoes would become significant competitors to German shoes. During his stop in Japan, he made contact with a Japanese shoemaker Onikusa. When Phil returned from his word trip with a contract to distribute Onitsuka shoes in the U.S., he started his legendary Nike journey.

In the early days of Nike, Cashflow has always been a bottleneck for its growth. Although the American venture capital was booming then, most of them were in Silicon Valley, far away from Nike’s headquarter in Portland. Besides, the Shoe business is not the high-growth field V.C.s were looking for. As a result, Nike had to grow from bootstrapping and from bank loans. For the first five years, Phil had to keep a day job to earn Nike cash and work on Nike in the evenings and weekends. Nike was continuously groaned by its bankers and almost fell into bankruptcy in 1975.  

Nike rides the tide of Globalization. From Nike, you could see how Globalization (particularly Japan) have profoundly influenced the United States in the 1970s. Nike was initially just a distributor of Onikusa in the USA. Later, when it started selling shoes, it relied on loans from Nissho, a Japanese trade company.

Flying geese paradigm

The history of Nike is also a history of supply chain outsourcing from the U.S. to east Asia. Despite that it is the most famous sports shoe brand globally, Nike itself manufactures nothing and entirely relies on a global supply chain, which gave Nike the edge over Adidas. Nike’s supply chain was first in Japan, then was moved to Taiwan and later to China. Nike is not alone. A lot of other American companies (such as Apple and Tesla) followed the pattern.

Japanese scholar Akamatsu’s came up with a concept of Flying geese paradigm for the phenomenon that the Asian countries would catch up with the West like flying geese because the production of commoditized goods would continuously move from the more advanced countries to the less advanced ones in the region hierarchy. 

The flying geese paradigm is the reason why there are so many economic miracles in east Asian in the past few decades (Japanese economic miracleMiracle on the Han RiverTaiwan Miracle)

Then why Asia? Some crucial reasons are the region’s social and cultural characteristics: hard-working ethic, Collectivism, and low labor cost (initially), which are somewhat very different and complementary with the Western culture. You could get more context for the difference in this excellent documentary American Factory.

However, the social and cultural characteristics that help the countries catch up with the West are a double-edged sword. Although the flying geese paradigm created economic miracles in those countries, it also makes them prone to the “technology snapshot trap,”— a phenomenon in which a society develops involutely in a “snapshot” of the outdated technology because it fails to learn from outside or generate innovation innately continuously.

For example, Japan developed an advanced automobile and Electronics industry in the 1970s but failed to lead the personal computer revolution. Korea and Taiwan picked up the semiconductor industry but missed the Internet. Recently, “Involution” also became a hot topic in China social media. More and more people complain that society starts to stagnate and more and people have to face more fierce competition on limited resources.

In east Asia, the working population suffers from severe over-working (e.g., 996Karoshi) in the catchup process. Over-work culture will prevent people from learning new things and reduces fertility rates, which will drive up labor costs and reduce the competitive advantage of society in the long run. The obedient culture in the region also reduces the diversity of ideas and disruptive innovation within itself.

Companies like Nike combine both ends’ advantages by leveraging the West’s marketing and sales creativity and delivering high-quality yet cheap products using the Asia supply chain. However, this fundamentally drives the tension on both ends and is the inherent reason for a couple of trade wars.

We are facing a dilemma for Globalization. American people complain about the loss of manufacturing jobs, and Asian countries complain that the West captures most profits. We are at the crossroad of deglobalization, and the pandemic adds fuel to the process. A healthy society needs to strive for the right balance between the two cultures. Both ends should learn more from each other and take the opportunity to transform their culture and industry structures.

Categories
English Newsletter

The Revenge of Apple to Intel

One hot topic recently is that Apple released its new ARM chips — M1. It is not the first time Apple designs chips — Apple has successfully designed chips for its iPhone and IPads. It is also not the first time Apple uses non-Intel chips in its Mac products — Mac had Intel cores only since 2006.

Then why is it important? In short, this is a declaration of war from Apple to Intel and a game-changer for Reduced Instruction Set Computer (RISC) in performance-sensitive applications.

What are Instruction Sets?

Developers use chip instruction sets to communicate with computer chips. Metaphorically chip instruction sets are similar to the alphabets of human languages.

There are only twenty-six characters in English, but more than three thousand in Chinese. Similarly, the size of chip instruction sets also varies. Reduced Instruction Set Computer (RISC) refers to building chips using a small instruction set. In contrast, Complex Instruction Set Computer (CISC) refers to the option that uses an extensive instruction set. (Please see here for more descriptions)

A little history

Early computer chips were all CISC and mostly were designed by Intel. In the 1980s, there was a movement of reducing the instruction set. The ARM technology was founded in this period, and Apple–IBM–Motorola alliance built the PowerPC chips for Macintosh computers.

On the other side of the table, the Windows-Intel alliance (a.k.a “WinTel“) kept investing heavily in CISC. The rest is history; WinTel crushed Apple computers in personal computing. Apple had to switch to Intel chips in 2006. ARM survived only in a then niche market of IoT devices thanks to its energy efficiency.

Then the mobile Internet era came, thanks to Apple’s iPhone release. ARM is appealing for those applications because people care about the battery life of smartphones. As a result, ARM captured 90% of the market share for mobile processors. Intel lost the mobile war because it suffered from the Innovator’s Dilemma and wasn’t willing to risk upsetting its existing CISC business.

Despite ARM’s success in mobile phones, Intel still holds the crown for applications that require high-performance. Many people think this is due to CISC’s inherent superiority in high-performance computation, and Intel is safe in those fields.

Apple declared this is wrong through the release of M1. Intel maintained CISC’s advantage in the high-performance applications through massive investment, and previously there was no significant player who could compete.

Except for Apple. Some early users mention the performance of M1 could be comparable to NVIDIA’s popular 1080Ti GPU. The TensorFlow team also shows new M1 chips could outperform many workstations for AI applications, which have the highest computation requirements. 

What’s more, Apple has a great track record for disrupting industries. A lot of ARM manufacturers will follow Apple’s path to optimize ARM for high-performance applications, and they are eager to do so, given that the mobile phone market is saturating.

Besides, NVIDIA now owns ARM. The merger gives both edges in the age of AI. The road ahead for Intel is not rosy. Would the aging Titan be able to hold its position? It’s hard to say. But one thing is sure. More competition in the field is a great thing for companies in downstream areas like Cloud and AI, which could benefit from increased computation powers and reduced cost.

The market share ARM in different fields

Note: There is an interesting podcast from A16z about Apple Silicon. 16 Minutes #46: Apple Silicon — A Long Game, Changing the Game

Note: Although it is very promising, please still wait for a few months before you decide to upgrade to Big Sur or M1 chip if you want to use it for ML training. A lot of the libraries are not compatible with the new system yet (tweet)

Categories
English

What AI practitioners could learn from Tesla

This is the second blog about Tesla, please also read the blog of The Rise and Fall of a Great Inventor if you are interested to learn more about Tesla’s life.

Tesla is one of the key figures in the early evolution of the electrical industry. Tesla has good showmanship and is very good at attracting public attention through jaw-dropping demos.  In one such public demo, Tesla ignited light bulbs using his body. Those demos helped Tesla raise funding for his Alternating-Current motors, which greatly extended the applications of electricity.

0_tc81proTFGVIpWqC.pngTesla’s Magnifying transmitter 

In Tesla’s later years, his focus shifted to wireless energy transmission. Tesla planned to set up a set of energy transmission towers in the world, and any person could receive energy through a hand-held device. It was a grand project. Tesla raised some initial funding from J.P. Morgan to implement a prototype. Unfortunately, an Italian physicist and radio pioneer Marconi finished the wireless telegraph across the Atlantic Ocean in 1901, which attracted most of the public attention and overshadowed Tesla’s work. What’s worse, Tesla spent all the funding to build a huge tower in Wardenclyffe but failed to deliver a workable solution. He was turned down when trying to request more funding from J.P. Morgan. He was never able to fulfill this dream for the rest of his life.

0_5ASkdPah11dXRjfb

1904 Image of Wardenclyffe Tower.

Although it happened one century ago, Tesla’s story is still very relevant in the contemporary world in which AI is the new electricity. As an AI practitioner, I think there are several lessons we could learn from Tesla’s experience.

First, even for a super ambitious project, it is still important to make sure there are reasonable deliverables in the process. An ambitious vision may be crucial to get the initial resources. But in order to keep the marathon running, it is always good to plan a sequence of deliverables throughout the journey. The anti-pattern of promising too much while delivering too little needs to be avoided. Tesla was a visionary Inventor, but he lacked the practical mindset to manage the expectation of investors and showing deliverables.

Second, it is super important to be mindful of the relevant opportunities and be flexible for the plan. The development of technology is never a linear process. Tesla’s technology was very similar to what Marconi used for telegraph across the Atlantic and Tesla had much more experience than Marconi. Why didn’t he become the inventor of the telegraph? He failed to realize another important application of his technology — information transmission — and went straight to the grand goal of wireless energy transmission. Had he realize that achieving wireless communication was equally important and may be helpful for his final goal, he probably would invest more in the direction. 

For AI, the 2016 game between AlphaGo and Lee Sedol played a similar role as Tesla’s public demonstrations. The game attracted huge public attention and made many people realize the potential of AI. Under this hype, a lot of companies were founded with super ambitious goals that require decades to fulfill. And a lot of investors invested without a good understanding of this. What’s worse, a lot of the companies didn’t set up reasonable deliverables in a typical cycle of an investment fund. When those investors realized this gap, they may pull back investments, which will make the industry enter another winter.

It doesn’t mean that we shouldn’t work on moonshot AI projects. On the one hand, a lot of advanced AI projects will and should take place in universities under public support. On the other hand, for AI moonshots that are done in companies, we need to balance the grand vision with concrete milestones that are associated with the company’s core business. For example, one of the fields that AI works very well so far is the recommender system (e.g. the algorithm behind and Youtube or Instagram Feeds). The main reason for this is that its deliverables are very quantifiable (e.g., improve the daily activities users by x percentage) and directly contribute to the core business of the company, which is crucial to ensure continual support. I hope other fields could also find a similar positive feedback loop. It won’t be an easy path, but it is something that industrial AI  practitioners need to figure out.

Categories
English

The Rise and Fall of a Great Inventor

Recently I finished the book “Tesla: Inventor of Electrical Age” by W. Bernard Carlson and I highly recommend this book. I will write multiple blogs about the book and this first one will focus on historical facts and my thoughts on the rise and fall of Tesla.

tesla.jpg

Nikola Tesla was born in 1856 in the Austra-Hungarian empire. He came to the US in June 1884 to work for Edison Machine Works and left the company after a short-stay of 6 months. Later, he was approached by businessmen Benjamin A. Vail and Robert Lane to form a company but the two persons abandoned Telsa after a year. In the fall of 1886, Tesla was rescued by two other business partners Peck and Brown, who underwrote Tesla’s efforts to develop inventions into practical devices. On July 7, 1888, Peck and Brown sold the Tesla patents to Westinghouse for a lucrative deal and Tesla started to serve as a consultant for Westinghouse. Tesla left Westinghouse in August 1889 and in 1891 Westington tore apart the contract with Tesla under the pressure of investors. From 1892, Tesla started to give consultation on the Niagara fall powerhouse project, which finished in 1895 and established Tesla’s reputation as one of America’s leading inventors.

During 1895 and 1898, Tesla investigated some other things, like X-ray and radio-controlled boats. In 1899 and 1900, Tesla stayed in Colorado to perfect the wireless transmission system. In Nov 1900, Tesla was able to meet with the most powerful man on Wall Street, J.P. Morgan, and convinced Morgan to loan him $150,000 to support his wireless network. Around the same time, the Italian inventor Marconi was also working on a competing technology. In Dec 1901, Marconi finished the transmission of Telegraphy through Atlantic. The loss of the competition with Marconi forced Tesla to bet all-in on an even bolder project of wireless transmission of power. After the project failed in 1905, the life of Tesla as a bold inventor came as an end. Tesla spent most of this remaining life as a recluse in a New York hotel and was forgotten soon.

Telsa’s inventor career could be divided into three stages: Rise, Plateau, and Downfall stages. The first phase (Rise) is from when Tesla started to work for the Edison company and ended at the time when his sponsor Peck died. In this stage, Tesla, as a young immigrant, challenged the industry with his innovative thoughts of AC transmission. Tesla built his reputation by showing many magical demonstrations of electricity. Despite the dramatic promotion, Tesla’s work at this stage was pretty practical.

After Peck’s death that marked the start of the second stage (plateau), Tesla struggled to create a project that had commercial potential. Different from the first stage, Tesla at this time had more resources and still did a lot of amazing demonstrations to the public. However, a lot of his efforts like X-ray and Radio-controlled boats went nowhere. The biggest achievement this time is Niagara fall. However, Tesla’s role in the project is only a consultant and this is more a continuation of his work in the first stage. The lack of evidence to bring his innovative ideas to concrete commercial success restricted his ability to find patrons. 

The third stage is when Tesla got into the building the wireless energy transmission. He was able to secure some funding from patrons (like Astron and JP morgan) to pursue his dream thanks to his fame. However, none of his patrons at this stage was as serious and devoted as Peck and Brown were. To make matter worse, Tesla had to make bolder and bolder claims because he couldn’t catch up with new innovators like Marconi. The claims he made (like establishing a global wireless energy network) eventually backfired and cost him the credibility that was essential for him to raise funding.

Like everything in this world, the rise and the fall of Tesla are likely to be caused by many factors. I want to mention two of them in this blog:

A fundamental reason is that Tesla changed from a challenger to be a defender.  The rise of Tesla is because of his great contribution to Alternating Current (AC) technology. When Tesla first came to the US, Direct Current (DC) was more popular because it had a headstart in both research and industry. Notably, Edison was an ardent supporter of DC. However, AC had technical advantages over DC for long-distance transmission of electricity. Edison probably also realized the potential of AC. However, as the stakeholder of the Edison electric company (later became GE), which had already invested heavily in DC, Edison had to defend his commercial interests. In contrast, As a penniless immigrant, Tesla had no such burden so he chose to focus on the less-popular AC technology. In addition, like Steve Jobs, Tesla had the ability to create a Reality distortion field around him and to change people’s views. For example, Peck and Brown initially wanted Tesla to focus on DC that already had a market. Tesla organized a dramatic demonstration of using AC to make a copper egg spin by themselves that turned Peck and Brown into ardent supporters of AC. Through those demonstrations and continuous improvement of the technology, Tesla successfully challenged the status of DC.

The situation became completely different when Tesla was competing with Marconi for wireless technology. Similar to Edison in the 1980s, Tesla has been blinded by the sunk cost. Tesla’s long-term success in using electricity as a medium of energy transmission made him unable to realize the significance of wireless communication. Although he did propose a plan of using his technology for information transmission, it was mostly a strategy to secure funding and energy was still the main focus. Eventually, Tesla failed in the competition because of the complexity of wireless energy transmission. The newcomer Marconi didn’t have this burden and set wireless communication as the primary focus from day one. 

Another reason is that Tesla couldn’t find another strong business partner to fill the gap after Peck’s death. It was Peck who helped Tesla set up the strategy of patent-promote-sell that Tesla used throughout his career. However, the recipe wouldn’t work without any of the three ingredients. Tesla is very good at innovating and patenting. However, Tesla lacks the business acumen to execute the promotion and sales strategies.

Undoubtedly Tesla has a great talent for showmanship. However, the key to promotion in this context is to establish credibility among the professionals and managers in the electrical industry, who are the decision-makers for Tesla’s patents. Peck knew it very well so he tried to secure the endorsement of Professor Anthony, a well-established figure in the community, as the first step of the promotion campaign. After Peck passed away, Tesla relied mostly on mass media, which eventually portrayed him as a magician instead of a serious inventor. The mass media coverage helped Tesla in the beginning but eventually backfired and made him harder to secure financial support. Tesla also lacked sales and negotiation skills. For example, Peck helped Tesla negotiate the deal with Westinghouse, which was very favorable to Tesla himself, but Tesla allowed Westinghouse to tear it apart in 1891 after Peck passed away. Later, Tesla negotiated a very unfavorable deal with J.P. Morgan, which allowed Morgan to take the majority stake without a clear clarification of Morgan’s duty in the partnership. The ambiguity eventually damaged the partnership and the deal became a blocking stone when Tesla tried to raise funding from other investors.

Despite the enormous legacy he has left us, Tesla was forgotten for a long time. It was only in recent years that he re-entered people’s attention thanks to the electric car brand named after him. Interestingly, the founder of Tesla Motors, Elon Musk, shares a lot of characteristics with Tesla. Both are bold innovators and are good at showmanship. As a great disruptive innovator, Tesla and his story are still relevant in our contemporary world. 

Categories
English

Why we need to fight unitedly and what we should learn from Wuhan

I know it is hard to be spared from covid19 news but please pardon me for another one. The virus, now officially declared as a global pandemic, has turned the world upside down in the past months. More than 200K people have been infected and 10K people have died, with the number drastically increasing day by day. It is a pity that the containment of the virus has failed in the western world, but we should still keep our chin up and fight unitedly to mitigate the consequences. As a person who grew up in Hubei, studied in Wuhan for college and now lives in the US, I have some words to share.

 

First, we should take this virus extremely seriously because people (both young and old) do die from it. There have been a lot of great articles about why social distancing is critical. The next couple of weeks would be critical for mitigating the consequence of the virus. Without strong measures, the number of infections increases by 10x every 8 days. What this means is that by April 19, there will be 90M infections in the US alone if no strong measure is taken. Please also refer to this great article for more information.

 

Second, we need to be strategical and to learn from past lessons. In particular, I want to share what happened in Wuhan in the early days of the outbreak. As the first location of the outbreak, Wuhan suffered a lot, but it looks like the western world has very little knowledge of the mistakes that were made in Wuhan. Here are some pieces of advice based on what I know:

 

  1. Keep calm. Don’t rush to hospitals. Stay away from panic groups. Use tele-diagnostics options if possible. In the early days of the Wuhan outbreak, a lot of people who got common code rushed to hospital but the waiting hours could be extremely long because the hospital system was crowded. Many people got infected in the tiring waiting process during which their immune system was impaired. Hospitals need to provide an appointment system to minimize the number of people who go to hospitals at the same time. People who have symptoms should trust the system and don’t panic.
  2. Staying at home is not sufficient, you need also protect your families, especially if you have any symptoms. The very first measure of Wuhan city government was to advise people to self-isolate at home. What happened after that? The situation got much worse because people who stayed at home infected their families. When young kids or old people got infected, then people had to visit hospitals in person, which transmitted the virus to other families. If you have symptoms, please distance yourself with your families and do whatever you can to minimize interaction with them. There would be some inconveniences but it would be life-saving. The government needs to take strong actions to provide shelters for people who have symptoms for isolation. People who have been in contact with infected persons or have symptoms should go to the shelters instead of staying at home. Extended testing needs to be made to guide who should stay at home and who should go to the shelter.
  3. Don’t think that only old people get the virus and young people don’t have responsibilities. A common misconception is that young people won’t get serious symptoms from the virus so won’t be in trouble. If the hospital is crowded, nobody could be spared. Besides, young people do die from the disease. Widespread of the virus also makes it more likely for older people to get infected and the virus may also mutate in the process.
  4. Most importantly, be gentle, nice and supportive to other people. I know this is a difficult time and a lot of people are frustrated, confused and panicked. However, as a human community, we shouldn’t point guns to each other during this difficult time. Certain government officials want to take advantage of the situation and promote their political agenda. This is NOT a time for that.

 

I want to end the article with a positive note. With a strong measure, we could reverse the course of the virus spread. Yesterday Wuhan has reported 0 new case. Wuhan people have achieved it with 2 months of efforts, so every country and community should be able to do the same. The critical thing is to stay united and learn from each other.

 

God bless every human in the world!

 

Categories
English

A Better Future for Everyone

I recently finished the book the war on normal people written by Andrew Yang, who is a democratic presidential candidate for the 2020 election. The normal people in the title refers to the silent majority of Americans who haven’t received higher education and have suffered in the recent technological development and globalization. As a “techie” who lives in Silicon Valley, I am very thankful for Andrew to share his thoughts and opinions. This book is a great way for people like me to have a realistic view of America, a view Andrew developed after he worked many years to foster entrepreneurship outside Silicon Valley.

My first question when I saw the title was: Aren’t we living in a peaceful time? Who is waging the war to normal people? According to Andrew, we are in a “war” created by the nature of capitalism to achieve efficiency and the new technology that favors high-skilled workers at the cost of the normal people. Andrew feels that normal people lack the ability to stand on their own foot in this “war.” As a result, the government needs to step in and to give normal people a hand. The solution Andrew proposed is the  the universal basic income (or so-called freedom dividend), which is to give $1000 to every American citizen. While there may be a better solution, Andrew has got the problem right — there is a massive job shift in terms of both skill requirements and locations. Unfortunately, not everyone could adapt to the shift. 

More than 5 million manufacturing workers lost their jobs after 2000, and it was the emotion behind this massive job loss that sent Trump to the white house in the 2016 election. In the first part of the book, Andrew focuses on where and why the jobs have gone. There are two reasons for the job loss — automation and globalization. The former allows employers to replace a lot of human workers with machines, the latter allows employers to outsource many jobs to countries of lower labor costs. The American workers now need to compete with both machines and cheap labors overseas.

The underlying driving force is the desire for companies to maximize profit for shareholders. In spite of Andrew’s grudges, I think private companies should not be scrutinized for this. After all, it is the same force that propels the machine of the market to operate and it is the government’s duty to set the rules of the market and private companies are not and should not be wealth fare programs.

Another point mentioned in the book is that fewer jobs are created by the new technology than those that were eliminated. The example it cites is that while Walmart employes over +1 million works, new tech giants like Google or Facebook employes a much fewer number of people to reach the same scale. Although arguably this is true, the point is less clear if we consider the total number of job opportunities they provide. Google or Facebook are also platform companies that enabled many more job opportunities. If you count contractors, gig-workers and content creators, the number of job opportunities created by these platforms is much larger than the number of full-time employees they hire. Besides, more than 4 million enterprises rely on Google and Facebook and they provide many more job opportunities. 

Although the total number may not change, there are still significant changes in jobs. On the one hand, new jobs require much higher skills than the ones they replace. On the other hand, New jobs appear in different geographical locations than the old ones.

While globalization and automation are shifting the nature of jobs, there is no mechanism to help people, especially less educated people, to adapt to the shift. Two decades have passed since the job losses started. As two decades is a long time and if the adaption still doesn’t happen, we couldn’t expect the trend will automatically be reversed. The losers are deprived of their representative rights in the market. They will eventually stop looking for job opportunities. 

It is right that the government needs to step in to help bridge the gaps. However, instead of distributing money unconditionally, the government should give stronger incentives and guidance for people to adapt to the change.

First, the government should invest in education and ensure it is affordable and accessible. Since new jobs generally require higher skills. Equal education is the key to ensure every person has equal access to the new opportunities created by technologies. STEM education in the US is considerably lagging. Student debt is a serious issue. The quality gap of education in public and private schools needs to be narrowed. Also, adult education should be subsidized by the government because, arguably, adults who decide to continue education need more incentives and encouragement than kids. 

Second, the government should help bridge the gap among different geographical areas. Currently residing in a place outside Silicon Valley and New York means a huge loss of job opportunities. When I graduated, I couldn’t find a good job in Boston so had to come to Silicon Valley. Most of my friends had similar experiences. Given that Boston is already a talent hub and a college city, the situations of other cities could only be worse. For this issue, I recommend another book The New Geography Of Jobs, which is authored by Enrico Moretti and also discusses the geographical shift of jobs in the united states. 

One joke is that why silicon valley becomes the tech hub is that VCs like the good weather of the area. Although this is a joke, it is certainly true that VCs have a preference for certain places over others. If VC lacks the incentives to go to other cities, the government should step in and either provide venture fundings or adjust the legislation to give VC more incentives to invest in other communities. 

At the same time, for those who want to relocate to a technological hub, the government should also provide the necessary help. This is tricky because we need to make sure this won’t cause too much pressure to the communities outside the existing technical hub. 

Although I don’t agree with the plan of giving freedom dividends unconditionally, I do think the government should play a more active and positive role to create a better future for every person in the upcoming new technological society. I am glad to hear that Andrew is on the way to find a solution and his campaign has already increased people’s awareness of the issue a lot. Best wishes to Andrew for his journey.

2019-12-15

Categories
English

Why people write less nowadays?

When I was a kid, I used to write a lot. I wrote diaries daily to snapshot my thoughts and articles weekly to express my opinions. Remember that this was the time when computers were rare and when I could only write using pens and papers.

Nowadays, I have much better writing equipment yet I am much less prolific. I have been thinking to write an article for a long time. Unfortunately, I have been procrastinating, and my last post has been a few years ago. I also observed similar trends among my close friends. I feel this is a contemporary trend. When was the last time you write a love letter, a travel diary? People write much less than before, and even when they write, most of the writings are utilitarian, i.e., they convey facts but no emotions. I am talking about the writings that force authors to squeeze every emotion and creativity from the bottom of their brain to touch the audience’s soul.

There are simply so many distractions nowadays. We are such a deeply connected species — smartphones feel like our external body parts and everyone is online 24×7. Smartphones are great for killing times but unfortunately not so good tools for writers. Computers are much more useful for report writing, which requires extensive searches of information, but less so for pure literature writing. Besides, more and more smartphone applications are grabbing people’s attention by dividing information into small pieces to fit people’s fragmented calendar. As a consequence, people are accustomed to superficial thinking and spend less time on meditative writing.

It doesn’t mean that there is nothing we could do. A simple anti-dose for this is more self-discipline. We are lured but thankfully are not forced to live in a fragmented way. Just block your calendar, lock your phone and pick up your pen from today. This is not a perfect solution but it works if executed well. Good luck and enjoy being a writer!

Jing Conan Wang

2019/11/23