Standardization of encryption algorithms since the turn of the millennium

Due to a significant change in the political situation in the 1990s, cryptography ceased to be the domain of spies, and the development of technology enabled the use of these approaches in science and commerce. In order to facilitate this made it necessary to ensure adequate standardization. This article seeks to summarize standardization activities.

Standardization activities in cryptography

During the late twentieth and early twenty-first centuries, several standardization competitions emerged which had an impact on cryptography and its implementation. Whether it be the DES algorithms (which originated significantly earlier) and AES, the hash function of SHA-1, SHA-2, SHA-3, or currently Post Quantum Cryptography or LightWeight Cryptography. All are reactions of industry and commerce to ensure confidentiality, integrity, trustworthiness and authenticity of communication.
In the field of standardization, the two leading countries from the Cold War era were the Western Bloc, represented by the US and its allies, compared to the Eastern Bloc and its satellites. Initially, they were algorithms designed for military communications and state secrets. After the moderation of international relations, these tools were used in commerce, industry and services. Unfortunately, at present, a significant number of organizations overlook the lessons of this era. More precisely, confidentiality does not ensure the integrity or authenticity of a message. Similarly, in the field of IT, the Kerckhoff rules are forgotten. One of them has been saying for over 150 years: "Cryptography must be safe and easy to use, must not require an increased intellectual burden".

USA

In the field of cryptography, standardization has a long tradition in the United States. An example is the famous "Rainbow series" of books on security (rainbow series), which subsequently developed into Common Criteria. A large number of these rules originated in the military and were published over the years, for example, by the NIST Institute, eventually they were standardized and became known at the federal level as FIPS.
The Standardization Institute cooperates with the NSA, which ensures the security of communications of a government organization. Unfortunately, the cooperation with this organization is not always perceived positively. There have been several problems in history that make the relationship to the NSA at least ambivalent (ECB mode, DES and 3DES, Dual EC_DRBG, Simon&Speck) and some procedures can be called kleptography. On the other hand, for the development of security and cryptography, despite all the aforementioned maladies, the NSA has done a huge amount of work.

DES (Digital Encryption Standard, USA)

One of the first standards was the DES algorithm (FIPS 146), which was developed at IBM. Horst Feistel was behind the development of this algorithm, who developed the original Lucifer algorithm as early as the second half of the 1960s. It was decided to publish it in early 1970, and since around that time there was a demand for an encryption standard, IBM and the NSA established a close collaboration. Thanks to this collaboration, the algorithm continued to evolve but there were also distinctive features of the algorithm. The result was the DES algorithm, which was subsequently standardized in 1977. Later, due to the key being too short, the 3DES algorithm was implemented. It used several passes through the DES algorithm, each with a different key. This resulted in a kind of extension of the key material either in the form of three encryptions and three keys, or two keys and the EDE mode (encryption-decryption-encryption). In 2004 the DES algorithm was declared obsolete, around 2017 all 64b algorithms were generally declared obsolete due to the Sweet32 attack. As of January 1, 2024, neither DES nor 3DES may be used anymore.

SHA (SHA0, SHA 1, Secure Hash Algorithm, USA)

This was not a competition, the SHA algorithm (SHA-0 / FIPS 180) was designed in 1992 by the NSA and standardized in 1993. Due to the vulnerabilities found, modifications were made to this algorithm and in 1994, the NSA proposed its successor SHA1 (FIPS 180-1 and later FIPS 180-2). This was standardized a year later. In both cases, the Merkle–Damgård architecture was used. The algorithm has a limit on the length of input and the hash does not contain information about this length, so it is possible to attack by extending the hashed text.

AES (Advanced Encryption Standard, USA)

Based on significant community development and criticism of the NSA's cryptographic algorithms, there was for the first time a public contest for a new standard called AES (FIPS 197). What was the cause of this award is likely to be a long discussion, but in any case it was an excellent move. There was competition between cryptologists from almost all over the world, the contest was practically free, the standard was promoted by the contest itself. At the same time, suspicion of a possible back door was effectively eliminated, as it is very difficult to push such things into existing algorithms. At the same time it was the first American standardization contest ever won by a non-American (Joan Daemen and Vincent Rijmen). NIST released an economic impact analysis in 2017, and the potential economic impact of the algorithm is estimated to be around $250 trillion. The list of contestants is in the following table:

SHA-2 (Secure Hash Algorithm, USA)

Again, this was not a competition, the SHA2 algorithm was designed by the NSA in 1999-2001 and standardized in 2002. At that time, the results of the AES algorithm competition were still pending and efforts were still underway to evaluate impact and safety, so the classical approach was probably chosen. As with SHA-1, this was the Merkle–Damgård architecture. The design developed the original SHA-1 design, extended the output length from 160b to 256/384/512b, tried to deal with the mistakes of the original design, and was published under the name SHA2 (FIPS 180-2). Unfortunately, there remained restrictions on the maximum input size, or the lack of influence of input text length on the output hash.

SHA-3 (Secure Hash Algorithm, USA)

Based on the success of the AES competition and the search for a new hash function design, a competition was launched for a new hashing standard. Existing versions of SHA1/SHA2 had some limitations that were pointed out in the community. For these reasons, an international competition was held again, running from 2009 to 2012. In year 2014 saw the publication of a new standard. The result was a hash function built on a new architecture, called like Sponge function.
How to imagine this "sponge" function? The base is the input memory. This contains a part, referred to as a reservoir, the second is inserted with the input texts. XOF (eXpendable Output Function) operations are performed over the input memory, that combine the contents of memory. If we imagine this on a model of a classic washing sponge and water container, always when soak the sponge in water, fill it with water - the input text. Its compression makes XOF and throws away part of the data, part always stays inside. After we transfer all the water (data) from the container with the sponge, we can really wring it out provide output hash.
The winner of this competition was the KECCAK algorithm, hereinafter referred to as SHA-3 (FIPS 202). Outside of this architecture, other interesting approaches were created, one of them was the HAIFA design, used e.g. in the Blake/Blake2/Blake3 algorithm. Next, it was the second competition won by a non-American team. Coincidentally it was the same team of winners as in the AES competition.

LWC (LightWeight Cryptography, USA)

The light cryptography competition covered specific requirements for encryption algorithms, designed for systems with limited resources. The term covers low performance or low processor capabilities, low memory, insufficient randomness inputs or missing cryptoaccelerators. It can be the Internet of things, industrial systems, payment or access cards, etc. Even these devices need to provide acceptable communication protection, but the available resources for standard algorithms may not be enough. For this reason, the contest was announced in 2018 and the winners were announced in 2023. A draft for the new standard is currently published. The winner of this contest was the ASCON algorithm (NIST SP 800-232), which provides adequate protection even for devices with such limited capabilities. Interestingly, this algorithm was also the winner of the European CAESAR competition (see below)

PQC (Post Quantum Cryptography, USA)

A competition for new standards for quantum computer-resistant cryptography has been running since 2017. This competition expanded in 2022 to include a branch known as the Additional Digital Signature Scheme, Currently the first cryptographic schemes ML-KEM (FIPS 203), ML-DSA (FIPS 204) and SHL-DSA (FIPS 205) are available. The FN-DSA (FIPS 206) standard is in the design phase. Although we have a large number of digital signature algorithms from both the main and the secondary standardization branch, unfortunately for the time being only a single algorithm for key negotiation is available. The search for alternatives in this area is currently one of the important goals of standardization institutions. More about the algorithms that are part of the above selections in a separate article.

Europe

NESSIE and eSTREAM were two consecutive competitions that had an interesting connection from the cryptography point of view. The reason is simple, the eSTREAM contest is the direct successor of NESSIE. The reason was the failure of all current cipher contestants in the NESSIE contest. In addition to the two ECRYPT contests, two others were held in Europe, the PHC community contests and the CAESAR contest.

NESSIE (New European Schemes for Signatures, Integrity, and Encryption, Europe)

The NESSIE project was announced in 2000 and closed in 2003. It was inspired by the AES contest and focused on finding new encryption algorithms for Europe. In the case of current algorithms, however, it encountered a considerable problem of insufficient level of development. Virtually all current ciphers were excluded during the contest and, based on the knowledge gained, the eSTREAM project was subsequently launched a year later. Thanks to this project, significant progress was made in the field of current ciphers. At the same time, the NESSIE project accepted the newly selected AES algorithm (the result of the US contest) for use in Europe, as well as the newly selected CAMMELIA algorithm (the result of the Japanese competition CRYPTREC).

eSTREAM (ECRYPT Stream Cipher Project)

The successor to the NESSIE competition was eSTREAM. It started its activities in 2004 and the results were announced already in 2008. It focused purely on the development and choices in the field of current algorithms. One of the results of this competition was the very popular SALSA family of algorithms (Salsa/12, Salsa20, ChaCha/12 and ChaCha/20). The Blake group of hash algorithms (architecture HAIFA) was subsequently created.

PHC (Password Hashing Competition, Europe)

For the purpose of password authentication, a separate database with data in open form was originally used, then the DES algorithm for symmetric encryption began to be used. At the same time, crypt interface was created, its first use is dated back to 1974. The use of hash functions for one-way authentication was a matter of the 1980s and 1990s. Their unidirectionality was an advantage over symmetric ciphers. Unfortunately, with their use, there were also attacks that targeted both the hash functions and the way they were used. Initially, it was a use for hash functions for rainbow table (Rainbow table). In this case, the computation occurs and thus the exchange of computational time for disk space, thus greatly speeding up the attack. Fortunately, the use of rainbow tables is limited to a narrow range of possible inputs and can be relatively easily prevented by using salt (a parameter generated for each individual password separately). Unfortunately, the hash functions are not perfect and it was necessary to come up with additional methods of protection that move the attacker beyond the area of attack profitability. The first step was compound hash functions (HMAC), followed by PBKDF2 above hash functions and finally a special class of slow hash functions was searched for.
The special class of slow hash functions is able to prevent the creation of rainbow tables and has increased demands on memory and processor. This increases the cost of the attack beyond what the attacker accepts. In 2013, a PHC contest was announced and closed two years later. The goal was to ensure standardization, eventually cooperation with the authorities came to an end. Nevertheless, a winner was found and generally accepted, others algorithms were found to be extremely promising. The full list is as follows:

CAESAR (Competition for Authenticated Encryption: Security, Applicability, and Robustness, Europe)

In 2012, a competition was announced and opened a year later for new algorithms capable of providing authenticated encryption. Developments in this area have shown some weaknesses associated with AE (Autenticated Encryption) and AEAD (Authenticated Encryption with Associated Data). This means encryption where it is possible to prove not only confidentiality, but also authenticity, i.e. who encrypted this data. This competition was closed in 2019, but its results did not come to fruition in the form of a standard. The competition itself is difficult to read. Nevertheless, the winner of this competition also won the NIST LWC competition. The CAESAR project list is here:

Japan

Japan has its own standardisation structure. It uses the CRYPTREC panel for selection, which also takes into account standards of other countries.

CRYPTREC (Cryptography Research and Evaluation Committees, Japan)

This is a Japanese commission that carried out the evaluation of mechanisms for the first time in 2003. This activity was repeated in 2013 and 2023 (it seems that every 10 years). The results of the evaluation of the individual encryption algorithms are published and create binding rules for the Japanese market. Moreover, the Camellia algorithm (one of the winners) is internationally accepted and partially accelerable using AES-NI instructions. The full list is here:

South Korea

Standardization in South Korea has taken a different path. This country is trying to promote its technologies through products manufactured on its market, while also supporting generally accepted standards. For this reason, the products of their companies also support the national algorithms ARIA (cooperation of security forces and academia) and SEED (security forces). Both of these algorithms are part of e.g. SSL/TLS. In addition to these algorithms, KpQC standardization for quantum computer-resistant cryptography is currently running, which is led by the world's leading experts.

KpqC competition (Korea Post Quantum Cryptography competition, South Korea)

This is an ongoing competition to select the corresponding algorithms for quantum computer-resistant cryptography. More about the algorithms that are part of the above selections in article

Russia

The Russian Federation still maintains the GOST standards (государственный стандарт / gosudarstvennyi standard) from the time of the Union of Soviet Socialist Republics. This is a set of standards originally corresponding to the requirements of the Soviet Union and to some extent the members of the RVHP group, currently rather the Euro-Asian Council for Standardization, Metrology and Certification, which operates within the Commonwealth of Independent States (CIS) and the Euro-Asian Economic Union (EAEU). At the moment there are standards, but I am not aware of any competitions in this area. Here, too, algorithms such as AES and others are accepted. Of interest in this area is the Magma algorithm, which is a peer of the DES algorithm.

India

India is planning a fairly extensive cryptography program, but unfortunately I have not been able to find any information regarding specific competitions if they are taking place. Roadmap is available here.

China

The ShāngMì family algorithms were designed and developed at the Center for Secure Communication at the Chinese Academy of Science (CAS) and the Cryptography Testing Center (Commercial Cryptography Testing Center, National Cryptography Administration). The goal was to design appropriate methods for security of communication with authentication for networks using SSL/TLS and WiFi networks, where a possible backdoor in existing technologies was a concern. At least some of these algorithms were declassified in approximately 2006 and became national standards in 2016.
I have added two algorithms to this list that do not belong here. These are ZUC algorithms, used in 3G/4G/5G networks and standardized 3GPP, as well as the SSFF3 algorithm. The ShāngMì family together with these algorithms is thus currently used in a wide range of applications, from WiFi, TLS, via access cards, communication with GPS Beida, mobile networks and possibly in other, special applications.

In January 2020, Chinese standards dedicated to quantum computer-resistant cryptography were published. At the same time, new symmetric algorithms were published. This material describes both symmetric and quantum computer-resistant algorithms. More about the quantum computer-resistant algorithms that are part of this selection can be found in article.

China is trying to demonstrate its independence in this area as well. Recently, I came across information that Russia should be interested in cooperating on this development, but the information is scarce and difficult for me to read. More information about the competition can be found at:
CACR cryptography standardization

Second group: uBlock, Ballet, FESH, ANT, TANGRAM, RAINDROP, NBC, FBC, SMBA, SPRING

Autor článku:

Jan Dušátko

Jan Dušátko has been working with computers and computer security for almost a quarter of a century. In the field of cryptography, he has cooperated with leading experts such as Vlastimil Klíma or Tomáš Rosa. Currently he works as a security consultant, his main focus is on topics related to cryptography, security, e-mail communication and Linux systems.