I don’t know how to get up, but I’m always in love.
Just like us.
Ten years ago, I saw it because I believed it.
Ten years later, in front of you, you see everything.
Yao Weibin (Wenjing), the technology leader of Aliyun CDN, who participated in the convoy of Shuang11, was touched by the words typed on the screen with a turnover of 213.5 billion yuan in 2018.
CDNIt was one of the first technologies to participate in Twin 11. After ten years of Twin 11, looking back on the past ten years, great changes have taken place in business scale, technical architecture and the engineers behind the scenes. Their stories are not as wonderful as those in Aliyun’s Crazy Men, but they are worth remembering.Remember!
Technical Helmsman Scene: Shuang11 is the most important thing for Technical Junior Two!
11At 00:19 on November 11, Wen Jing saw the small partners in the battle room talking easily. He knew that the flood peak of Shuang11 had passed smoothly. He began to brush Taobao. He bought lobster at the end of the ceremony 24 hours later.
Unlike the anxiety and excitement of previous years in Guangming Ding (Shuang11 Operational Command Room), more than 100 members of Wenjing Team have been calm and calm this year. They are distributed in different offices and cooperate online in five locations. In the context, business has driven technological development and personal growth in recent years.。 The stable system, mature combat plan and tacit teamwork of Shuang11 in the past ten years are the indelible marks of this team.
（Wen Jing buys babies at Shuang11 venue.
As the leader of CDN technology, Wenjing is not the most “old” classmate in the team, but he has experienced the rapid development of Taobao business for several years, witnessing every step of technological transformation from the initial service of Taobao and CDN within the group to the cloud CDN which serves the largest number of customers in China today.
Wenjing is a very “tortuous” student. He goes to the girls’dormitory to help tune up the network and follow the tutor’s “private work” of system development. He prefers to acquire knowledge from the exercise of projects rather than from books. Slowly, he came into contact with embedded development, wireless sensor networks, LinuX, Nginx, Squid, to the server development transformation. What really makes him feel the technical value of being an engineer is after doing open source. “There were many open source modules, some with thousands of STAR on Github, and the code I wrote could help thousands of people.Developers, even foreign developers will write me a thank-you letter, which not only improves the technology, but also gives me a strong sense of identity, but also because of open source, let Ali see me. After joining Ali, Tengine, in charge of Wenjing, once ranked the tenth Web Server in the world.。
Fortunately, he caught up with Shuang 11 in his first year in Ali. For Taobao technicians at that time, Shuang11 was the most important thing every year. “For us, the most important thing is that at zero point, the system has to carry, the less the downgrading measures the better.” Wen Jing recalls that the technical principles at that time were as follows:First, we need to estimate the business, the number of QPS concurrently issued by tens of millions of people, and whether the system can support it. Second, we need to make technical plans, and how much capacity all depend on the system, and how to deal with the problems once they occur. After the first few years of double 11 test, the system has gradually been able to be positive in the extreme situation.The ability of regular operation has been greatly improved in terms of high performance and stability. Later, our technological capabilities have far exceeded the scale of the second kill, and the current technological architecture has been enough to support, all technologists are at ease. “
Setting sail: From beginners on the road to running all the way
Li Ke, a senior technical expert, has been involved in Shuang11 for ten years. It seems like yesterday when we talk about the winter of 2009. “Everyone is a novice, how to prepare, how to cooperate, are basically touching the stones across the river, CDN used the hardware solution at that time, performance and flexibility are still very limited.System. Nobody expected that the speed of e-commerce business development was so astonishing. Some of our nodes were on the verge of collapse. At that time, it really depended on the manual treatment of several operation and maintenance classmates. That is, after Shuang11 that year, Taobao Core System Department announced the establishment of self-research CD.N is on the agenda.
（Discussions with “comrades-in-arms” at Shuang11 escort site
Of course, the accumulation of technology is not achieved overnight. Every year we forecast the peak in the second year, but each time we underestimate the explosive growth of e-commerce. Under such high-intensity business pressure, almost all bottlenecks will be encountered by us, memory, CPU, IO, network, back and forth.Many rounds of software optimization and node architecture upgrade have been done. For example, for overload, the long front-end and back-end connection mechanism is added so that each connection can serve multiple requests. In order to reduce IO overhead, a consistent HASH strategy is used to unify the disks of multiple machines. Take memory – & gt; S on a single computer.SD – & gt; SATA’s multi-layer storage strategy can dynamically transfer hot and cold resources to improve disk efficiency. In order to cope with peak pressure, we will prepare business lock volume, node pressure measurement, downgrade plan, purchase temporary bandwidth and other measures in advance. In those days when our monitoring and data were not perfect,Shuangxi relies on each person to open multiple terminals and stare at a number of nodes. Once the machine load is found to be too high, manual intervention is carried out immediately.
In this way, the team relies on screen-staring + manual operation to survive the past few years. The CDN architecture has been customized and optimized, and the team’s technical strength has been rising in the midst of many challenges.
In a twinkling of an eye, Wenjing believes that this year is critical to CDN: “First, we decided not to use external commercial CDN, because we found Taobao traffic, commercial CDN is not enough to support. The reason is that Taobao was China’s first website with so many pictures, and then the SSD machine.The machine has not been so popular, most of the CDN manufacturers are still using SATA machines, when the volume of data surge, its IOPS can not run, the node will immediately IOB hang up. Second, when most CDN manufacturers are using Gigabit machines, Taobao CDN is already using Gigabit machines. We are in the middle of the process.China’s first manufacturer to use all SSD machines, and completed 40G nodes. At that time, the technology architecture was very advanced.
Evolution: in the days when opportunities and challenges coexist
2013With the increase of CDN capability, many businesses have begun to use CDN, such as secondkill, red envelope, details page, etc. CDN can offload a large number of requests and bandwidth, to avoid the inadequate capacity of the source station services unavailable. Visibility says: This year we have technically achieved static and data.Through static and dynamic separation, static content is abstracted, cached on CDN, and dynamic part is acquired asynchronously through ESI backsource. After the static transformation, Shuang11 has carried tens of times the peak traffic.
At the same time, CDN’s digitalization has begun to produce more and more abundant internal business. The first version of the data platform can provide five-minute granularity data for each domain name, including business data, such as hit rate, bandwidth, QPS, and monitoring data, such as RT.T, packet loss rate, etc. CDN farewell to the era of big pot rice, began to refine the operation, which also provides the basis for the commercialization of CDN.
Until 2014, CDN not only supported Shuang11 within the group, but also began to serve external customers. It is clear to me that in Shuang11 this year, there was a traffic speculation within the team. Engineers designed a prediction system for this purpose. Through the historical data of Shuang11 each year, and the prediction and summation of business lines.Actual situation, real-time forecast of late peak traffic, used to adjust resources in time, such as 11 a.m. on the same day to accurately figure out how many T the night peak.
2015The year 2000 also means a lot to the CDN team, because they almost recited 3.25 in that year.
At that time, Taobao went to the whole station HTTPS, but because of the previous pressure measurement and the evaluation of new services did not achieve the goal, the computing overhead of the whole station HTTPS led to CPU, memory began to become bottlenecks. “When Shuang11 started, everyone else was cheering for data innovation, but our team’s engineers watched.Watching the memory of the online node exploded and restarted, I was standing on the customer site at that time. I really had a dead heart. Wen Jing recalled.
At that time, the CDN team had already arrived in Aliyun, often running all over China, going deep into the customer site and providing close service. The focus of the whole team was no longer just to plan and guarantee for Shuang11. “We’ve gone from pure technicians to architects who need business thinking, using CDN to help customersTo solve problems. It is precisely because of this, this experience has sounded an alarm bell for us, after each new business was born, we have made sufficient plans, resolutely avoid repetition, let alone allow this situation to occur in customers.
（CDNTeam 2017 double 11 group photo left three empty right three Shihao right six scenery)
In 2016, the emerging live and short video services are developing rapidly, and the bandwidth is growing very fast. Zhou Zhe (Shihao), a senior technical expert, believes that the team has developed a high-performance, low-latency and stable caching system after several years of double-11 entrance examination. We are able to serve such large-scale pictures well.Business, video business, of course, is no longer the case.
“But when the large-scale VOD business came up, I realized that the understanding of VOD scene was insufficient. Video on demand (VOD) has higher requirements for the first screen, the Carton experience, and the overall hit rate and the return protection. Therefore, we will improve the whole system through link monitoring, user experience-based adjustment.Degree, improve the overall cache hit rate, TCP stack optimization, to improve the video playback experience.
The original CDN team focused on the network, pulled up the brothers team, carried out a technical fortification for half a year, polished the scheme of live full link guarantee. This scheme has also been successfully applied to the Tianmao Shuang11 evening party. It’s still clear that TFBoy appeared when the team was in the Shuang11 evening party of the Aerospace Protection Team in 2016.On the screen, the bandwidth soared several times, “This is the first time I’ve seen the power of small fresh meat!”
At the same time, Youku joined the Ali family, which brought CDN a lot of business scenarios of large file acceleration and video acceleration, so that the whole team’s resource construction and technical level have been supplemented. At the technical level, it also strengthens the CDN team’s understanding of video business scenarios and improves the real-time transcapsulation of video formats.Related technologies such as whole network anti-theft chain, content copyright protection, P2P, etc.
Since 2017, Aliyun CDN has embarked on the road of internationalization, carrying out large-scale construction of overseas nodes to meet the needs of domestic and overseas users. At the same time, after the 2008 Russian World Cup, Spring Festival Gala and other major events, the team size of CDN,Customer scale and technical strength are advanced together. The automation, intellectualization, refinement and service capability of the whole system are all increased. This team, which can win the battle, is already very confident and calm when facing Shuang11 again.
Today: programmers spend double 11 together
After ten years of accumulation and precipitation, Wen Jing thinks Shuang11 this year can be said to be a harvest year.
CDNThe team mainly supports e-commerce and cat night, the whole process can be said to be as smooth as silk, zero fault and zero carton. At the same time, cat evening live broadcasting provides 4K video with ultra-high definition and high frame rate. Through real-time frequency processing and transcoding, cat evening live broadcasting can further improve the quality of the picture and make the netizens have enough eye addiction.
CDNFor Youku 25 million users to provide video distribution, the entire platform’s QPS has also broken through the historical peak. During the whole service process, depending on the whole link quality monitoring and intelligent automatic scheduling, the duty personnel can control the whole situation only by paying attention to the data market.
During Shuang11 this year, Wenjing’s team also provided accelerated protection for Lazada’s e-commerce CDN, and combined with live solutions, helped Lazada push the Shuang11 evening party to Thailand, Malaysia, and reveled in Shuang11 with Southeast Asians.
“The dream of drinking tea and purchasing double 11, which was once a luxury, has finally come true this year, and 213.5 billion miles has also been contributed by our staff on duty,” he said half-jokingly and half-proudly.
（After the zero flood crest passed smoothly, the programmers finally had the heart to buy it.
Along the way, Wen Jing once wondered: Why can’t we extend the time span of the activity to three days, or one week? So maybe we don’t have to take such a big risk to bear the daily peak pressure of more than 10 times! However, when he came to the present, he realized deeply that it was precisely because of the concentrated outbreak.The business peaks, again and again, are driving technology forward, which has created today’s technology team that can win. Technology should thank the business!
Ten years Shuang11 is like a vast history of Ali’s technological development. In this way, CDN uses its unique ups and downs to add a strong ink. Looking forward to the next decade, we will continue to follow each other along the way.
（See for the tenth time at Shuang11 escort site: How many decades have you lived?
This article is the original content of Yunqi Community, which can not be reproduced without permission.