Reinforcement learning for energy-efficient control of multi-stage production lines with parallel machine workstations
Alberto Loffredo, Marvin Carl May, Andrea Matta
download PDFAbstract. An effective approach to enhancing the sustainability of production systems is to use energy-efficient control (EEC) policies for optimal balancing of production rate and energy demand. Reinforcement learning (RL) algorithms can be employed to successfully control production systems, even when there is a lack of prior knowledge about system parameters. Furthermore, recent research demonstrated that RL can be also applied for the optimal EEC of a single manufacturing workstation with parallel machines. The purpose of this study is to apply an RL for EEC approach to more workstations belonging to the same industrial production system from the automotive sector, without relying on full knowledge of system dynamics. This work aims to show how the RL for EEC of more workstations affects the overall production system in terms of throughput and energy consumption. Numerical results demonstrate the benefits of the proposed model.
Keywords
Artificial Intelligence, Sustainability, Manufacturing Systems
Published online 9/5/2023, 9 pages
Copyright © 2023 by the author(s)
Published under license by Materials Research Forum LLC., Millersville PA, USA
Citation: Alberto Loffredo, Marvin Carl May, Andrea Matta, Reinforcement learning for energy-efficient control of multi-stage production lines with parallel machine workstations, Materials Research Proceedings, Vol. 35, pp 428-436, 2023
DOI: https://doi.org/10.21741/9781644902714-51
The article was published as article 51 of the book Italian Manufacturing Association Conference
Content from this work may be used under the terms of the Creative Commons Attribution 3.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
References
[1] Bipartisan Policy Center. Annual energy outlook 2022. EIA, Washington, DC, 2020.
[2] Renna, P. and Materi, S. (2021) A literature review of energy efficiency and sustainability in manufacturing systems. Applied Sciences, 11(16), 7366. https://doi.org/10.3390/app11167366
[3] Mouzon, G., Yildirim, M.B. and Twomey, J. (2007) Operational methods for minimization of energy consumption of manufacturing equipment. International Journal of Production Research, 45(18-19), 4247-4271. https://doi.org/10.1080/00207540701450013
[4] Duque, E.T., Fei, Z., Wang, J., Li, S., Li, Y., 2018. Energy consumption control of one machine manufacturing system with stochastic arrivals based on fuzzy logic, in: 2018 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), IEEE. pp. 1503-1507. https://doi.org/10.1109/IEEM.2018.8607749
[5] Frigerio, N., Cornaggia, C.F. and Matta, A. (2021) An adaptive policy for on-line energy-efficient control of machine tools under throughput constraint. Journal of Cleaner Production, 287, 125367. https://doi.org/10.1016/j.jclepro.2020.125367
[6] Jia, Z., Zhang, L., Arinez, J. and Xiao, G. (2016) Performance analysis for serial production lines with Bernoulli machines and real-time WIP-based machine switch-on/off control. International Journal of Production Research, 54(21), 6285-6301. https://doi.org/10.1080/00207543.2016.1197438
[7] Cui, P.-H., Wang, J.-Q., Li, Y. and Yan, F.-Y. (2021) Energy-efficient control in serial production lines: Modeling, analysis and improvement. Journal of Manufacturing Systems, 60, 11-21. https://doi.org/10.1016/j.jmsy.2021.04.002
[8] Loffredo, A., Frigerio, N., Lanzarone, E., Matta, A., 2021. Energy-efficient control policy for parallel and identical machines with availability constraint. IEEE Robotics and Automation Letters. Vol. 6(3), pp 5713-5719 https://doi.org/10.1109/LRA.2021.3085169
[9] Loffredo, A., Frigerio, N., Lanzarone, E., Matta, A., 2023. Energy-efficient control in multi-stage production lines with parallel machine workstations and production constraints. IISE Transactions. https://doi.org/10.1080/24725854.2023.2168321
[10] Loffredo, A., May, M.C., Schäfer L., Matta, A., & Lanza G. Reinforcement Learning for Energy-Efficient Control of Parallel and Identical Machines. CIRP Journal of Manufacturing Science and Technology. Under Review.
[11] Sutton, R.S., and Barto, A.G., 2018. Reinforcement learning: An introduction. MIT press,
[12] Panzer, M., Bender, B., 2022. Deep reinforcement learning in production systems: a systematic literature review. International Journal of Production Research 60, 4316-4341. https://doi.org/10.1080/00207543.2021.1973138
[13] Baer, S., Turner, D., Mohanty, P., Samsonov, V., Bakakeu, R., Meisen, T., 2020. Multi agent deep q-network approach for online job shop scheduling in flexible manufacturing, in: Proceedings of the 16th International Joint Conference on Artificial Intelligence, pp. 1-9.
[14] Stricker, N., Kuhnle, A., Sturm, R., Friess, S., 2018. Reinforcement learning for adaptive order dispatching in the semiconductor industry. CIRP Annals 67, 511-514. https://doi.org/10.1016/j.cirp.2018.04.041
[15] Malus, A., Kozjek, D., et al., 2020. Real-time order dispatching for a fleet of autonomous mobile robots using multi-agent reinforcement learning. CIRP annals 69, 397-400. https://doi.org/10.1016/j.cirp.2020.04.001
[16] Kuhnle, A., Schaarschmidt, M., Fricke, K., 2017. Tensorforce: a tensorflow library for applied reinforcement learning.
[17] Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P., 2015. Trust region policy optimization, in: International conference on machine learning, PMLR. pp. 1889-1897.
[18] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017. Proximal policy optimization algorithms.
[19] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M., 2013. Playing atari with deep reinforcement learning.
