Tech Hammer

That's right, when Ning Wei looked at Huo Zhiheng's subtitles in the video and listened to his powerful voice, a practical Internet technology derived from the process of solving N-S equations - turbulence algorithm - suddenly appeared in his mind.

This feeling is very strange.

It was as if the words of the video host in the video kept inspiring him, and then a complete algorithm structure was displayed in his mind combined with the knowledge he had learned before.

Then came another feeling of inspiration.

If I didn't record this inspiration, Ning Wei felt that I would go crazy immediately.

So he jumped out of bed.

"What's wrong, baby Ning, are you scared to death by my literary talent?"

"Bah, you guys are talking about your own business, don't pay attention to me, I have an idea that I want to record!" Ning Wei said to Xu Ruixuan casually, then sat down in front of his computer, quickly opened the computer, and then clicked on the browser.

The first thing he had to make sure was that the turbulence algorithm derived from the N-S equation that suddenly appeared in his mind had indeed not been invented yet.

Soon, Ning Wei was sure that it was indeed not the case!

Because if this turbulence algorithm had come out, everyone would not still be at a loss to deal with those malicious web crawlers and could only deal with it through patchwork.

That's right, the turbulence algorithm that popped into Ning Wei's mind at this time was an intelligent algorithm specifically designed to deal with malicious web crawlers.

In the Internet era, the network is flooded with various web crawlers.

The so-called web crawler is actually a program and script that automatically crawls network information in batches according to certain rules. The most widely used application is probably search engines. Through web crawler technology, these search engines will enter the collected information into the database at a certain frequency through these web crawlers to ensure that users can get more information when retrieving from the engine entrance. Find what you really want from the vast information on the Internet more accurately.

This type of web crawler is generally called a universal web crawler.

The other type is focused on crawlers.

This is a technology that continuously crawls specific websites or specific information.

Generally speaking, there is no problem if you can use this technology in compliance with the robots protocol.

But in fact, many web crawlers all over the Internet are not doing human work.

The case of web crawlers that is closely related to everyone is scalpers grabbing votes.

For example, 12306 is the most visited website by various web crawlers in the world.

Every holiday, especially during peak travel periods such as May Day, National Day, and Spring Festival travel, train tickets in popular areas are always particularly hard to get, and can even be sold out in seconds. Moreover, it is difficult to open the 12306 software at this time, basically thanks to this kind of web crawler.

According to statistics, the peak number of clicks on Huaxia 12306 reached 5.9 billion times per hour, with an average of more than 1.6 million clicks per second. Obviously it is not a number that normal users can generate.

The official program also uses various upgraded verification codes to prevent scalpers from grabbing tickets. For a period of time, those abnormal-level verification codes were even difficult for an ordinary person to know what to do.

Even though the function of combining people and tickets is now launched, countless ticket grabbing software still provide this service by purchasing ticket grabbing packages at a higher price.

In addition, major airlines are also the hardest hit areas for illegal web crawlers, especially those airlines that often release special fares. Almost every time a special ticket is released, it will be sniffed by this kind of crawler, and then directly booked without paying.

For airlines, generally there will be half an hour for the buyer to pay after booking such special tickets. If they do not pay within half an hour, they will enter the ticket pool again. However, crawler technology can make these special tickets enter the ticket pool. It was snatched up again within 0.01 seconds until the scalpers found a buyer willing to increase the price and used the buyer's identity information to purchase the ticket and pay.

It is conceivable that both 12306 and major airlines hate this kind of crawler.

After all, they don’t make a penny from the scalpers’ price increase, and it also increases the network load.

In particular, major airlines originally intended to pass profits to customers and increase customer stickiness, but as a result, customers did not enjoy low-price tickets, and may be annoyed because they paid high prices and did not enjoy corresponding services.

In addition to this kind of web crawler, there are also some more malicious web crawlers that silently lurk in the network and continue to crawl all kinds of private user identity information, major hotel chains, various APP user systems, etc. , are the targets of these web crawlers.

Unfortunately, there are currently no very good defensive technical means against these malicious web crawlers, and they generally rely on various hardware and software firewall technologies to isolate them.

What makes countless developers even more embarrassed is that security and convenience often cannot be achieved at the same time.

This is just like those crushing verification codes that 12306 once introduced. While it temporarily stopped the crawlers from wreaking havoc, it also made countless ordinary people confused.

At this time, the turbulence algorithm in Ning Wei's mind was able to solve the problem of malicious crawlers while taking into account convenience.

To express the function of this algorithm in understandable language is probably that the steady-state data flow will flow slowly and normally like running water through the server data interface. Each connection request will directly affect this steady-state data flow. It's like the calm flowing river surface becomes turbulent due to the small fish swimming upstream.

When the server is configured with the turbulence algorithm, and by upgrading the verification system, the steady-state data flow can intelligently determine whether various connection requests are normal or other illegal requests through countless accesses, and thereby determine whether web crawlers When doing data crawling, it is still normal access by normal customers.

After making the distinction, the algorithm can automatically direct these crawlers to the target directly into a data turbulence, where these crawlers can only crawl all kinds of chaotic and complicated invalid data and then feed it back to the crawler author.

If this process is carefully arranged, it may directly affect the security of the device receiving the information, leaving these malicious crawlers with nowhere to hide.

It sounds simple, but this turbulence algorithm is not simple. It contains many underlying neural networks and deep learning algorithms. For example, it almost simultaneously uses recurrent neural networks, generative adversarial networks, deep shrinkage networks, various regressions, etc. .

There are also algorithms deduced from the N-S equation.

Soon, Ning Wei roughly recorded several parts of the algorithm on the computer.

Then the C language environment was called up.

It’s not that I don’t want to use python. After all, python is extremely simple compared to C. There are many packages that can be called directly. Just like a person who has never learned to cook, if he uses a cooking package, he can make extremely delicious food with just a microwave oven. meals.

But Python cannot meet Ning Wei’s needs, not to mention that many parts of the turbulence algorithm are not included in Python at all.

Moreover, the school offered a C language class in his freshman year, and Ning Wei’s grades were pretty good.

Of course, this result is good, not at all what Ning Wei is showing now.

For example, when he was learning the C language, the pointers and linked lists that gave him a lot of headaches are now as if he has mastered them without a teacher, and he can use them with ease.

This is probably a growing pain!