Using real-time manufacturing data to schedule a smart factory via reinforcement learning

https://doi.org/10.1016/j.cie.2022.108406Get rights and content

Highlights

  • Design a cyber-physical architecture and scheduling mechanism for smart factory.

  • Establish a genetic-programming-based scheduling rule library.

  • Develop a module to realize the dimension reduction and clustering of data.

  • Use RL to train the decision-making agent for appropriate rule selection.

Abstract

Under the background of intelligent manufacturing, internet of things and other information technologies have accumulated a large amount of data for manufacturing system. However, the traditional scheduling methods often ignore the production law and knowledge hidden in the manufacturing data. Therefore, this paper proposes a cyber-physical architecture and a communication protocol for smart factory, and a multiagent-system-based dynamic scheduling mechanism is given using contract net protocol. In the dynamic scheduling mechanism, the problem formulation module and scheduling point module are designed first. Then, a genetic programming (GP) method is proposed to form sixteen high-quality rules, which constitute the scheduling rule library. Meanwhile, combining with autoencoder, self-organizing mapping neural network and k-means clustering algorithm, the state clustering module is designed to realize the efficient clustering of production attribute vector. Moreover, an improved Q-learning algorithm is used to train the GP rule selector, so that the decision-making agent can choose the appropriate GP rule according to the production state at each scheduling point. Finally, the experimental results show that the proposed method has feasibility and superiority compared with other methods in real-time scheduling, and can effectively deal with disturbance events in the manufacturing process.

Introduction

With the increasingly fierce competition in the global market, manufacturing enterprises are faced with more stringent requirements in improving production efficiency, improving product quality, reducing resource consumption and reducing production costs (Dai et al., 2019). In order to enhance the competitiveness of manufacturing industry and expand the international market, countries have formulated corresponding national strategies, such as Industrial Internet in the US, Industry 4.0 in Germany and Made in China 2025. These strategies can help industry realize smart manufacturing through the application of various advanced technologies in the manufacturing field. Among them, the rapid development of Internet of things (IoT) and other new information technologies provides rich real-time data for manufacturing system (Bueno et al., 2020). According to statistics, in the modern industrial system, the total amount of data generated by mechanical equipment, production management, application services, etc., is more than 1000 Exabytes every year (Yin & Kaynak, 2015), which forms an explosive growth. By analyzing and processing these data, the manufacturing system can obtain valuable knowledge and make wise and forward-looking decisions, so as to achieve a positive impact on the production of enterprises (Cui et al., 2020). In other words, data-driven manufacturing is gradually becoming the main means to realize smart manufacturing.

From the perspective of the whole industry, big data is becoming more and more important in the manufacturing industry. It will play an important role in the fourth industrial revolution, that is, bringing intelligence to manufacturing. The first three industrial revolutions led to the steam era, electrification era and information era respectively, which make production become large-scale and automated. Germany calls the fourth industrial revolution Industry 4.0 (Olsen & Tomlin, 2020). Industry 4.0 uses cyber physical system (CPS) to digitize the supply, manufacturing and sales information in production. It aims to improve the intelligent level of manufacturing industry and build a smart factory with adaptability, resource efficiency and genetic engineering (Nakayama et al., 2020). In the smart factory, various manufacturing resources interact in real time using communication protocols and standards. Meanwhile, sensors and automatic identification technology are used to collect data at each stage of the product lifecycle (Li et al., 2015). These data include user data, supplier data, equipment data, order data, etc. Through the analysis of these data, enterprises can achieve lots of goals (Usuga Cadavid et al., 2020), such as predicting product price trend, improving product design, reducing energy consumption, reducing production cost and providing intelligent maintenance service. To sum up, big data technology has changed the traditional mode of product manufacturing and management, which makes the production process transparent. It overcomes the problem of lack of coordination and information sharing in the traditional supply chain, and avoids the bullwhip effect, thus realizing the objective of providing intelligent application services using smart manufacturing.

In recent years, many scholars have conducted relevant research on the big data-driven manufacturing. Majeed et al. (Majeed et al., 2021) proposed a framework of big data-driven sustainable and smart additive manufacturing, and it can help additive industry leaders to make better decisions at the beginning stage of product lifecycle. Tao et al. (Tao et al., 2018) discussed the role of big data in supporting smart manufacturing and proposed a conceptual framework for data-driven smart manufacturing. Ku et al. (Ku et al., 2020) developed a solution to support traditional industries to adopt smart manufacturing and empower digital transformation. Mörth et al. (Mörth et al., 2020) introduced a design perspective for IoT-driven analytics in intralogistics to optimize the internal logistics systems. Tao and Qi (Tao & Qi, 2019) proposed a new information technology driven service-oriented smart manufacturing framework to facilitate the visions of smart manufacturing. Qu et al. (Qu et al., 2019) summarized the evolution, definition, objectives, three requirements and components of smart manufacturing system, and proposed an autonomous model driven by dynamic demand and key performance indicators. Subramaniyan et al. (Subramaniyan et al., 2020) developed a data-driven approach to diagnosing throughput bottlenecks, using the combined knowledge of the maintenance and machine learning (ML) domains. Zhang et al. (Zhang et al., 2020) proposed an integrated framework to holistically describe the active discovery and optimal allocation of smart manufacturing services. Li et al. (Li et al., 2019) developed an uncertain learning curve and some useful theorems by utilizing uncertainty theory, and applied it to the single-machine optimization problem. Cheng et al. (Cheng et al., 2020) constructed a high-quality training dataset and proposed an adaptive ensemble model to achieve fast and accurate makespan estimation.

As the core of intelligent manufacturing, production scheduling can significantly improve the decision-making level of shop floor system and realize the efficient operation of manufacturing execution process. The traditional scheduling methods mainly include heuristic algorithm and metaheuristic algorithm. Heuristic algorithms assign priority to manufactured objects, thereby implementing scheduling and allocation. It mainly includes shortest processing time, longest processing time, first in first out and other scheduling rules. Meta-heuristic algorithms are intelligent optimization algorithms, which can find high-quality scheduling schemes through a large number of iterations and searches. Wu et al. (Wu et al., 2021) established a mixed integer linear programming model for the scheduling problem in three successive processes in car production, and used several meta-heuristic algorithms to find good solutions. Huang et al. (Huang et al., 2021) proposed an improved iterative greedy algorithm based on the groupthink for solving the distributed assembly permutation flowshop scheduling problem. Zhu and Zhou (Zhu & Zhou, 2020) proposed an efficient evolutionary multi-objective grey wolf optimizer for the flexible job shop scheduling problem with job precedence constraints. Yuan et al. (Yuan et al., 2021) proposed an improved nondominated sorting genetic algorithm for the resource scheduling problem in intelligent manufacturing workshop. Chen et al. (Chen et al., 2021) proposed six constructive heuristics and an iterated greedy algorithm for the distributed blocking flowshop scheduling problem.

However, the traditional scheduling methods often ignore the information accumulated by the manufacturing system, resulting in the production law hidden in the manufacturing data not being mined. At the same time, in the actual production workshop, disturbance events often occur, such as machine breakdown and new order insertion. These disturbance events make the original scheduling scheme no longer have superior scheduling performance, and even make the original scheme infeasible. In the current smart factory, the new information technology can quickly perceive, collect and control the manufacturing information of various resources in the production system. Therefore, it is of great significance to study manufacturing data-driven dynamic scheduling under the framework of intelligent manufacturing. At present, more and more scholars try to combine industrial big data with scheduling optimization. Wang et al. (Wang et al., 2019) proposed a new multiagent-based real-time scheduling architecture for an IoT-enabled flexible job shop, which uses real-time data to deal with unpredictable exceptions. Wang and Gombolay (Wang & Gombolay, 2020) developed a novel graph attention network-based scheduler to automatically learn features of scheduling problems towards coordinating human-robot collaboration in real-time environment. Hu et al. (Hu et al., 2020) proposed an adaptive deep reinforcement learning-based automated guided vehicle (AGV) real-time scheduling approach with mixed rule for the flexible shop floor to minimize the makespan and delay ratio. Caldeira et al. (Caldeira et al., 2020) proposed an improved backtracking search algorithm for the flexible job shop scheduling problem considering new job arrivals. Kong et al. (Kong et al., 2021) established a novel energy- efficient rescheduling model with Time-of-use energy cost and applied a variable neighborhood search algorithm to obtain near-optimal solutions.

It can be seen from the above that in the intelligent manufacturing environment, it is of great significance to study the data-driven scheduling method for workshop. A reasonable manufacturing paradigm can provide a technical framework for data-driven manufacturing, and various advanced technologies make data-driven manufacturing possible. However, there are still few literatures on manufacturing data-based production scheduling, and few studies take into account the redundancy of production characteristics. Therefore, combined with the effective manufacturing information processing technology, it is possible and meaningful to develop a real-time data-driven dynamic scheduling method for smart factory on the basis of the advanced data acquisition hardware and artificial intelligence (AI) method. This is the main motivation of this paper.

From the perspective of dynamic scheduling method design, heuristic algorithm (i.e., scheduling rule) has strong real-time performance and is suitable for dynamic scheduling. However, a single scheduling rule cannot maintain high-quality scheduling performance in the face of orders with different sizes. Therefore, the ideal situation is that at each scheduling point, the workshop selects an appropriate rule from the rule library according to the real-time manufacturing data, so as to realize efficient production operation. Based on this idea, this paper takes real-time manufacturing data as state and scheduling rule as action, and then uses reinforcement learning (RL) algorithm to design a dynamic scheduling method for smart factory. This is the method design idea of this paper.

In order to address these challenges, this paper proposes the cyber-physical architecture for smart factory, and design the smart-factory-oriented dynamic scheduling mechanism based on multi-agent system (MAS) and contract net protocol (CNP). The dynamic scheduling mechanism includes five key modules. Firstly, the problem formulation module is proposed for modeling the smart factory scheduling problem. Secondly, the scheduling point module is designed for agent to determine whether a time point meets the requirements of scheduling point. Thirdly, the genetic programming (GP) method is designed to generate sixteen high-quality rules and form the scheduling rule library. Fourthly, combining with autoencoder, self-organizing mapping (SOM) neural network and K-means clustering algorithm, the state clustering module is proposed to realize the efficient clustering of manufacturing data. Fifthly, an improved Q-learning algorithm is used to train the GP rule selector, so that the decision-making agent can select the appropriate GP rule according to the production state at each scheduling point. Finally, the experimental results show that the proposed method has feasibility and superiority in real-time scheduling, and can effectively deal with disturbance events in the production process.

The rest of this paper is organized as follows. Section 2 generally describes the smart factory for intelligent scheduling. Section 3 presents the five key modules in MAS. In Section 4, the performance of smart factory is evaluated through training experiment and comparison experiments. In Section 5, the conclusion and future work are given.

Section snippets

Cyber-physical architecture

In order to realize the intelligent scheduling, a cyber-physical architecture is established for smart factory to execute the effective management based on real-time manufacturing data, and it is shown in Fig. 1. Smart factory is divided into the physical world and cyber world.

The physical world is composed of lots of manufacturing resources, including the machines, automated guided vehicles, automated storage and retrieval system (ASRS), etc. Through the combination of mechanical equipment,

Problem formulation module

Using the problem formulation module, the model of production scheduling problem in smart factory is given below. There exist a set of a machines M = {Mx, x  = 1, …, a}, a set of b transportation equipment TE = {TEy, y = 1, …, b}, a set of m jobs J = {Ji, i = 1, …, m}, and each job Ji has a set of ni operations Ji = {Oij, j = 1, …, ni}. Each operation can be processed on different machines, and the processing time and cutting power of an operation on different machine candidates are different.

Platform and case introduction

Based on the manufacturing platform in a university laboratory, we built the smart factory system to verify the performance of proposed method. Meanwhile, in order to execute the training and performance comparison, some experiments are carried out in the simulation environment. It is a workstation equipped with Windows 10 64 operating systems, 16G RAM, Intel i9 3.50 GHz CPU.

The parameter settings of production scheduling problems are given in Table 4. In each scheduling problem, new orders are

Conclusion and future work

In order to realize the data-driven manufacturing, this paper proposes the cyber-physical architecture for smart factory, and uses CNP to design the MAS-based dynamic scheduling mechanism, which includes the five key modules: problem formulation module, scheduling point module, scheduling rule library, state clustering module and GP rule selector. Comprehensive experiments are conducted, and the results show that the decision-making agent can select the appropriate GP rule according to the PAV

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported by the National Science Foundation of China (No. 51875171), the General program of Natural Science Foundation of Jiangsu Province (BK20201162) and the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX21_0465).

References (34)

  • Dai, M., Tang, D., Giret, A., & Salido, M. A. (2019). Multi-objective optimization for energy-efficient flexible job...
  • S. Yin et al.

    Big Data for Modern Industry: Challenges and Trends

    Proceedings of the IEEE

    (2015)
  • Cui, Y., Kara, S., & Chan, K. C. (2020). Manufacturing big data ecosystem: A systematic literature review. Robotics and...
  • T.L. Olsen et al.

    Industry 4.0: Opportunities and challenges for operations management

    Manufacturing and Service Operations Management

    (2020)
  • Nakayama, R. S., de Mesquita Spínola, M., & Silva, J. R. (2020). Towards I4.0: A comprehensive analysis of evolution...
  • J. Li et al.

    Big Data in product lifecycle management

    International Journal of Advanced Manufacturing Technology

    (2015)
  • J.P. Usuga Cadavid et al.

    Machine learning applied in production planning and control: A state-of-the-art in the era of industry 4.0

    Journal of Intelligent Manufacturing

    (2020)
  • Cited by (6)

    View full text