Code optimization is very important, especially for dynamic language like python. Recently I finished a python program of Approximate Dynamic Programming. One part of the program was to use dijkstra algorithm to calculate the shortest path distance in the network. Since the network is really large(about 1 million node), the first version of code was really slow which took about 2000 seconds each run. After code optimization(it took me about 6 hours), each run just takes 0.2 second. This is incredible improvement.
I used the the following tools during the process, which I highly recommend:
- python cprofile module –> this is a builtin profile module
- runsnakerun –> this is a viewer of the cProfile output
- line_profiler –> this is line by line profiler
The usage of runsnakerun is quite easy, just use the following two commands
$ python -m cProfile -o <outputfilename> <script-name> <options>
$ runsnake <outputfilename>
line_profil\ is a line by line python profiler and its webpage is:
add profile decorator to the function you want to optimize
$kernprof.py [-l/–line-by-line] script_to_profile.py
$ python -m line_profiler script_to_profile.py.lprof
Here are some tips to improve python performance:
1. in operator is really slow when the list is large. You may often use
if a in S:
When S is very large, thousands, for instance, in operator will become a bottleneck.
A good way to optimize this is have list with binary value to indicate whether a element is in the set or not. Then the code will be:
do some other thing
Of course, it will use more memory.
2. Call the library function as much as possbile.
Most of the library functions are highly optimized, or implemented in C++. So if you can call library functions, don’t implement it by you self in python. The for loop in python is slow. You can use scipy vectors to accelerate the program. You should try your best to reduce the for loop in python.
3 Don’t check each time for rare case.
you may have list with 100 elemnent, you will receive a index may or may not within range(0, 100). If it is in the range return the corresponding value, otherwise return -1. but 99.99% of the time the index is in the range. The bad is:
if idx < 1000:
B = A[idx]
B = -1
A better approach is:
B = A[idx]
B = 1
Be careful that don’t use broad Exception, use specific type of Exception.
This is the reading digest of Henry Ford’s <My life and My work>. Henry Ford is among the group of able people who completely change the transportation of human beings and consequently shape our society. He is a business man with great vision of the society. The real business leaders are always visionary and think big, that’s the reason why they are charismatic.
The Government is a servant and never should be anything but a servant.
Note: The government is not necessarily to be brilliant or productive by itself. The duty of Government is to make sure the most brilliant people in the society, no matter whether they are scientists, musician, entrepreneurs, or others, can have a safe and inspiring environment to use their brilliance.
Business men believed that you could do anything by “financing” it. If it did not go through on the first financing then the idea was to “refinance.” The process of “refinancing” was simply the game of sending good money after bad. In the majority of cases the need of refinancing arises from bad management, and the effect of refinancing is simply to pay the poor managers to keep up their bad management a little longer. It is merely a postponement of the day of judgment. This makeshift of refinancing is a device of speculative financiers. Their money is no good to them unless they can connect it up with a place where real work is being done, and that they cannot do unless, somehow, that place is poorly managed. Thus, the speculative financiers delude themselves that they are putting their money out to use. They are not; they are putting it out to waste.
Note: It is worse for the “good” money to be put into “bad” business than the “bad” money to be put into “good” business. By putting your money into a poor-managed company, you are actually endorsing the low-efficient production unit, which takes resources that should be available for more efficient business. The right approach to the problem of business to change its management, or simply shut it down and start over if management reform is not possible, but not to extend its life by burning more money.
George B. Selden, a patent attorney, filed an application as far back as 1879 for a patent the object of which was stated to be “The production of a safe, simple, and cheap road locomotive, light in weight, easy to control, possessed of sufficient power to overcome an ordinary inclination.” This application was kept alive in the Patent Office, by methods which are perfectly legal, until 1895, when the patent was granted.
Note: Any law can be misused, including the patent law. One thing need to be remembered, a good business needs patent to protect itself, not to attack others.
I believe that there is very little occasion for charity in this world–that is, charity in the sense of making gifts. Most certainly business and charity cannot be combined; the purpose of a factory is to produce, and it ill serves the community in general unless it does produce to the utmost of its capacity.
Note: Charity itself should not be abused. 99% of the charity in the world is just to satisfy people’s need of superiority, and it encourages nothing but laziness. The real charity is to provide the essential opportunity for poor people to earn their life by themselves.
The habit of failure is purely mental and is the mother of fear. This habit gets itself fixed on men because they lack vision. They start out to do something that reaches from A to Z. At A they fail, at B they stumble, and at C they meet with what seems to be an insuperable difficulty. They then cry “Beaten” and throw the whole task down. They have not even given themselves a chance really to fail; they have not given their vision a chance to be proved or disproved. They have simply let themselves be beaten by the natural difficulties that attend every kind of effort.
Note: It is just a matter of habit.
A country becomes great when, by the wise development of its resources and the skill of its people, property is widely and fairly distributed.
Note: The problem of China is not ideology. We don’t care about whether China is socialism or capitalism. What we care is whether people have the opportunity equality in the sense that the self-motivated people who want to earn their lives can get the resource they should have.
Perhaps no word is more overworked nowadays than the word “democracy,” and those who shout loudest about it, I think, as a rule, want it
Note: Look at Chairman Mao. Simply don’t trust politicians who shout loudest for democracy
The workingman himself must be on guard against some very dangerous notions–dangerous to himself and to the welfare of the country. It is sometimes said that the less a worker does, the more jobs he creates for other men. This fallacy assumes that idleness is creative. Idleness never created a job. It creates only burdens. The industrious man never runs his fellow worker out of a job; indeed, it is the industrious man who is the partner of the industrious manager–who creates more and more business and therefore more and more
Note: idleness create no value
The public was paying, and business was booming, and management didn’t care a
Note: 公司人数的快速增长并不代表公司的快速增长。The growth of payrolls does not necessarily mean the growth of the business. 我不明白为什么有的公司领袖会夸耀自己公司的人数规模，在我看来，创造了同样价值但是雇佣了更多的人并不值得夸耀。这不是创造了就业，这是降低了社会的效率。
An employer may be unfit for his job, just as a man at the lathe may be
Note: In the case that boss is unfit for his position, there are two options to increase the welfare of the society. The first option is to fire the boss and the second option is to fail the business. If the boss of the business is fired, there is some possibility that the business will recover, however the possibility is really really small, if not none. A more common case is that a business will be kicked out and a new company will replace it. The company is dead, long live the market. 我们总有一种情怀觉得要让给公司越长越好，公司寿命长对于这个社会并不是一定有益处的。公司的快速迭代是更好的让社会前进的选择。
I pity the poor fellow who is so soft and flabby that he must always have “an atmosphere of good feeling” around him before he can do his work. There are such men. And in the end, unless they obtain enough mental and moral hardiness to lift them out of their soft reliance on “feeling,” they are failures. Not only are they business failures; they are character failures also; it is as if their bones never attained a sufficient degree of hardness to enable them to stand on their own feet.
Note: I cannot agree more. If you can work only when you feel good, you are doomed to be a loser. Edit
I am thrilled that my project of “network malware simulation” has been accepted by Google Summer of Code 2012! I will work on this project from May to August, and a new interesting open source software will be born in this summer 🙂
click here for my proposal
In the 19th century, engineering was almost a synonym of mechanical engineering. A typical way to solve problem at that time was analyzing the mechanical structure, building the machine with gears and wrench and letting steam engine to drive the machine.
Steam engine is a revolutionary innovation as it introduced a “paradigm” for problem solving. If you have a mechanic description of the solution for your problem, steam engine can take care of the rest. There were tons of derivative innovation following this “paradigm”, such as plane, steamship, submarine, automobile and so on. (some of them used internal combustion engine instead, an improvement of steam engine). Theoretically steam engine and its derivatives can solve any mechanical problem, what people need to do is to describe the problem in mechanical language.
Nowadays people have been accustomed to resort to computers when they have problems. Computer is another revolutionary innovation because it provides a similar paradigm. The paradigm for computer is to formulate a arithmetic model, propose an algorithm correspondingly and run the algorithm with computers. A computer is basically a calculator that can do addition very fast. Since all arithmetic operations can reduce to to additions, theoretically computer provides us the ability to solve any arithmetic problems, what people need to is just to describe the problem in arithmetic language.
Despite their different appearance, steam engine and computer share some common points. Both of them provide a good solution for a fundamental problem, steam engine deals with the problem of “generate rotary movement with strong force” and computer deals with the problem of “do addition with fast speed”.
When a problem can be divided into a large sequence of fundamental problems, we can get the solution of this problem by solving those fundamental problems. With this in mind we can easily conclude there are a lot of other paradigms except for those described above. The figure shows the paradigm for steam engine and computer.
The reason why we need paradigm is that paradigm can save us time. The real world is too complex and we cannot do everything well. A reasonable way is to solve a small set of problems perfectly and transform the problem we want to solve into these problems.
Limitations of Paradigm
Every paradigm has its own limitation, which is determined by the two factors. The first one is the fundamental problem itself, you cannot use steam engine to read this blog since the transmission of bits cannot be modeled as a sequence of physical movement.
The second factor is our ability to solve fundamental problems. Doubtlessly computer is the most awesome invention we human beings have ever had. However, there are still many things that computers cannot do in spite of the efforts in the past 50 years. They cannot understand imprecise information, they have very limited communication skills, and so on. All these problems belong to the field of artificial intelligence. The reason is that although theoretically there exists an arithmetic model for all these problems, the number of arithmetic operations is beyond the ability of any existing computer even for a small problem.
In the dawn era of computers, scientists estimated that it should take only 10-20 years for machines to catch up with humans in terms of intelligence. However, the expected date was missed again and again. There was no big improvement in the recent 30 years and it seems that all researchers are just waiting computers to be stronger and stronger under moore’s law.
However, I strongly believe most of the scientists are in the wrong track. The paradigm of computers cannot provide us a perfect solution for artificial intelligence. Human intelligence cannot be described in arithmetic model with appropriate complexity. We need other types of paradigm to get perfect solution of artificial intelligence.
Jing Conan Wang
wangjing A_T bu.edu
Henry Ford <my life and my work>
Steam engine makes it possible for machines to move by themselve. Later the internal-combustion engine took the place of steam engine, but steamer was the beginning of this revolution.
If you want to install ubuntu in Macbook pro. plese refer the following webpage
be attention that, if you want to have the right resolution, download the driver from the nvidia offical site and don’t use the nvidia-current in apt-get, it will crash the GUI.
if you have installed nvidia-current and cannnot enter the GUI, just delete /etc/X11/xorg.conf and then type startx in the command window. then you will enter a GUI with low resolution interface. Then download the nvidia driver from official site and reinstall again.
I have no class this term so have more time for my hobby– entrepreneurship and music. I attended a technology lecture in last Wednesday and a one-day technology workshop in last Saturday.
The lecture was given by Brendon Howe, a BU alumni and the vice president of NetApp. Inc. a NASDAQ-listed storage company. To be honest, it was a little bit boring.
Mr Howe shared us his experience in both BU and NetApp. Since Howe grew up in the eastern coast, while NetApp Inc is a typical silicon valley company, I asked him about his opinion about the difference of east coast and west coast. I also ask whether he felt there is disadvantage for east coast graduates to find a job in high tech industry compared with west coast graduates as most of tech companies are silicon valley based.
His answer was that there are some difference, but the difference is not as large as people imagine. A east coast graduate student can easily adapt to the life in silicon valley. For the second question, he admitted that many companies are more willing to recruit local students, however, since globalization is the future trend, more and more companies set up their offices in the east coast, so this gap should be more or less bridged.
In terms of Saturday technology workshop, it was pretty much a MBA lecture. The workshop was organized by ITEC, acronym of Institute of Technology Entrepreneurship and Commercialization which is a very active technology organization under the school of management . About 30 students, mostly engineering and management students, joined the workshop. The students were divided into 5 groups to brainstorm biz ideas. Eventually each group will focus on one biz concept and try to draft a biz plan and give eight elevator pitch or biz presentation.
Our group focused on a gaming device in which you can control the avatars using your brain directly and watch game scenes through displaying glasses. It is actually a very interesting idea which can revolutionize the way people interact with gaming device. I eventually gave a elevator pitch. It was actually challenging for me since elevator pitch request very fast response which is harder for even native speakers. I tried to speak more passionate to overcomes the shortcoming of language and it seemed to work well.
For the music, I watched a very funny opera named secret marriage in Friday night. It was performed by BU students and free to all BU students, actually there is another good Boston Symphony Orchestra at the same time so I can only choose one. Choosing the opera turned to be not a bad idea. It says a young couple hide their marriage to the father of the wife. However, the betrothed husband of the wife’s sister falls into love with the wife, which make things complicated. The couple conspired to escape but was found. Eventually the young couple got permission of the wife’s father and show ends happily.
The only unhappy thing in the last week was research. The simulation results turned out to be crappy, so I switched to a new method, unfortunately the new method didn’t work, too. I stayed up last night but still didn’t get the result for meeting with advisor today, so the meeting is a little bit embarrassing as I made no progress in the past week. Anyway, research is only a small part of life.
Last Saturday, I attended the Sloan Asia Business Conference. Here is a brief summary of the conference.
The opening remark was given by Yasheng Huang, who is a faculty at MIT Sloan school of management. He just had a controversial talk about the development of India and China recently in TED2011 held in Edinburgh. In the talk, he attributed the success of China compared with India to the Human Capital—Better Literacy Ratio, Higher Life Expectation and less systematic discrimination of women. He also points that it is not fair to compare with India with China, which is a superstar in terms of economic development. Actually the performance of India actually is quite impressive among the emerging countries, while the slow movement of political reform in China may potentially hinder the economic growth in the future. The link of the talk is: http://www.ted.com/talks/lang/en/yasheng_huang.html
Lenovo Chief Marketing Officer. David Roman
As the chief marketing officer of Lenovo Corporation, David Roman introduced the global campaign named “For Those Who Do” that aims to promote the visibility of of the “Lenovo” brand outside China.
Despite the sluggish PC-market worldwide, Lenovo maintains 2-digits growth in the past several quarters and jumped to 2nd largest PC Maker, passing Acer and Dellconsecutively. But an embarrassing fact is the although the compony is a household brand in China, few people in western world know it in spite of its ThinkPad Series.
It is actually a common problem for many Chinese companies–so called the invisible Giants. Few Westerners know Baidu, SinoPec, ICPC, yet they are actually real mammoths even in western standard.
Now Lenovo’s focus is its ThinkPad Series, which formerly belonged the IBM. The deal was unbelievable then, since IBM PC department was triple the size of Lenovo at that time in terms of revenue.
It took Lenovo several years to learn how to be an international company. Now it seems that it has learnt the trick. In terms of my personal experience in United States, Lenovo is the most well-knowned Chinese Brand in United States. There is still a long way for other Chinese brands to go in United States.
SVP Finance & CFO of Global e-Commerce and Global Sourcing. Andreas Schulmeyer
This is the keynote I like most. The reason I favor this keynotes towards others is that it contains a lot of data rather than objective opinions. One figure is the comparison of the total population and GDP of China and United States. The left figure is for population and the right figure is for GDP. I guess the y axis is the GDP per capica. An interesting phenomenon is that the US’s value is also low in 1960s. The data actually needs to be validated.
Customer relationship Online and offline. There are five most components in culstomer Relatinship..
1. Social Network 2. Content 3. User Experience of Online Shopping 4. Brand Name 5 Search Engine.
Another interesting figure is the growth of US and China’s retail section.
Yuan wang, a high-level official in Chinese Development Bank and a consutant of world bank, gave a keynote about Chinese Economic situation. Her talk was a little bit boring. As a representation of new-generation Chinese policy maker, Wang reflected the impression of Chinese officials on westerners–well-educated, with international view yet still have bureaucratic ways of doing things. Most of the new Chinese policy makers,especially in economic field, receive well education. Some of them have western background and can speak pretty good English. However, different with their counterparts in Western Countries, Chinese policy makers always give people a feeling of commanding. Maybe it is caused by the booming Chinese Economy. All the investors worldwide want to go to China for new opportunities, and as the door-keeper of this huge market, the Chinese economic policy makers take it granted for the decisive power they have to let foreigner go or not.
Allan Kwan is a partner of Oak Investment. Now he focus the Investment on Northern Asia region, mostly in in Internet and food & beverage Industry. Before joining the Oak Investment, he worked in Motorola and Yahoo! consecutively. In Yahoo,
he took charge of the acquisition of 3721 founded Hongyi Zhou, which turned out to be a disaster, and the investment of Alibaba. What interests me is that his original major was mechanical engineering. His trace is quite typical and a lot of Chinese Students follow this way. Go to U.S for a Engineering Degree and then switch to financial field. He is a quite nice guy and gave two keynotes. In the second keynote, he noted the case study of LianLian Pay, one company he has invested in China. He used this case to illustrate the difference between business in China and that in America.The biz model of Lianlian pay is to construct the selling network of pre-paied card in China. Different with U.S, all the phone bills are charged using pre-paied card–user use to buy a pre-paid charging card and use the password in the card to charge their account. Lianlian pay construct the selling networks by setting partnership with thousands of grceries and convience store around China.
At the same, Quora emerged as a giant in the Q&A field. In Quora, people are pushed to the front, facing directly with humans instead of machines. Although the search in Quora is still about topic, not people. It is in the right direction. Give people more incentives by promoting interaction and acknowledgement.
After Page enrolled as CEO, Google has cut off a lot of programs, including the famous Google Lab. Google need to be more focused to win in the social war with Facebook, and all the distractions need to be cleared away, including Aardvark.
Of course, it’s reasonable for the benefits of Google. My point is that if you really want to make your idea reality, just don’t sell yourself to a big company. It makes no sense to expect big company to make your idea reality. I don’t means they will surely not, but just don’t keep high expectation on that since they have more “important” things to do. Also don’t expect you can make your idea reality within a big company, you are forced to do more “important” things under that environment.