Image by breakstock/Shutterstock

Researchers at University of Tsukuba have unveiled a groundbreaking advancement in data compression technology that could revolutionize how we handle data. They have developed a novel method that identifies, compresses, and fully restores recurring data patterns by processing the data once through a single compressor pass. This breakthrough technology achieves a compression ratio of 10%-30% higher than that of existing methods, marking a significant leap forward in data compression. The new method holds the potential to substantially reduce power consumption in next-generation data transmission systems for the Internet of Things (IoT) and cloud computing.

Tsukuba, Japan—Information and communication technologies, such as IoT and cloud computing, are cornerstones of modern society. The efficiency of these services depends heavily on the speed of processing and transmitting digital data, including image and sensor data. For instance, systems that identify individuals from surveillance camera images must streamline the entire process, from image transmission and artificial intelligence (AI) recognition to delivering results to users. Since data transfer often consumes more time than AI computation, there is a critical need for a data compression method that minimizes both data volume and communication time while preserving the integrity of the original data. The new technology can accelerate data transmission for AI computations and reduce the volume of data transmitted over wide-area networks, such as 5G/6G wireless communications and the Internet, resulting in significant global power savings.

Conventional data compression technologies compress and store data in finite chunks. For this purpose, they require bulky memory and processors that increase the size of the compressor and limit real-time compression to finite data streams. Researchers at the University of Tsukuba have addressed these challenges by developing a technology that automatically detects frequently occurring data patterns and compresses them to a minimum of one bit after a single pass through the compressor. This technology ensures complete real-time data restoration. Unlike previous technologies that can compress only one unit of data to one bit, the new method simultaneously compresses multiple data units to one bit, thereby enhancing the compression efficiency by 10%-30% compared to conventional methods.

In addition, this innovative technology enables the development of high-speed, compact data compression modules in hardware without the need for additional processors, memory, or other devices. If integrated into semiconductor chips and AI systems, this technology could revolutionize data transmission by boosting speed, reducing data volume, and conserving power in communication pathways, propelling us toward the vision of Society 5.0.

###
This research was funded in part by JSPS KAKENHI (Grant Number 23H03359), JST PRESTO (Grant Number JPMJPR203A) and JST AIP Acceleration Research (Grant Number JPMJCR24U4).

Original Paper

Title of original paper:

Universal Adaptive Stream-based Entropy Coding

Journal:

IEEE Access

DOI:

10.1109/ACCESS.2024.3429389

Correspondence

Associate Professor YAMAGIWA Shinichi
Institute of Systems and Information Engineering, University of Tsukuba

Related Link

Institute of Systems and Information Engineering