An Asynchronous LLM Architecture for Event Stream Analysis with Cameras
DOI:
https://doi.org/10.5281/zenodo.13639724Keywords:
llm architecture, stream analysis, pixel, camerasAbstract
Event-based cameras, as bio-inspired vision sensors, record intensity changes asynchronously. The Dynamic and Active-pixel Vision Sensor (DAVIS) enhances information diversity by combining a standard camera with an event-based camera. However, current methods analyze event streams synchronously, contradicting their nature and introducing noise. To address this, most approaches accumulate events within a time interval to create synchronous frames, wasting sensitive intensity changes. This paper introduces a novel neural asynchronous approach for event stream analysis. Our method asynchronously extracts dynamic information by leveraging historical motion information and critical features of grayscale frames. Extensive experiments demonstrate our model’s significant improvements over state-of-the-art baselines.
Downloads
References
Liu, Xiaoyi, & Zhuoyue Wang. (2024). Deep learning in medical image classification from mri based brain tumor images. arXiv preprint arXiv:2408.00636.
Chen, M. (2021, December). Annual precipitation forecast of Guangzhou based on genetic algorithm and backpropagation neural network (GA-BP). in International Conference on Algorithms, High Performance Computing, and Artificial Intelligence (AHPCAI 2021), 12156, pp. 182-186). SPIE.
Gu, Wenjun, et al. (2024). Predicting stock prices with FinBERT-LSTM: Integrating news sentiment analysis. arXiv preprint arXiv:2407.16150.
Yan, H., Wang, Z., Xu, Z., Wang, Z., Wu, Z., & Lyu, R. (2024). Research on image super-resolution reconstruction mechanism based on convolutional neural network. arXiv preprint arXiv:2407.13211.
Wang, Randi, & Vadim Shapiro. (2019). Topological semantics for lumped parameter systems modeling. Advanced Engineering Informatics, 42, 100958.
Wang, Randi, Vadim Shapiro, & Morad Mehandish. (2024). Model consistency for mechanical design: Bridging lumped and distributed parameter models with a priori guarantees. Journal of Mechanical Design, 146(5).
Qiu, Ri-Zhao, et al. (2022). Real-time semantic 3D reconstruction for high-touch surface recognition for robotic disinfection. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.
Wu, X., Wu, Y., Li, X., Ye, Z., Gu, X., Wu, Z., & Yang, Y. (2024). Application of adaptive machine learning systems in heterogeneous data environments. Global Academic Frontiers, 2(3), 37-50.
Chen, M., Chen, Y., & Zhang, Q. (2021). A review of energy consumption in the acquisition of bio-feedstock for microalgae biofuel production. Sustainability, 13(16), 8873.
Qiu, Ri-Zhao, et al. (2024). Feature splatting: Language-driven physics-based scene synthesis and editing. arXiv preprint arXiv:2404.01223.
Wang, Z., Yan, H., Wang, Y., Xu, Z., Wang, Z., & Wu, Z. (2024). Research on autonomous robots navigation based on reinforcement learning. arXiv preprint arXiv:2407.02539.
Jiang, L., Yu, C., Wu, Z., & Wang, Y. (2024). Advanced AI framework for enhanced detection and assessment of abdominal trauma: Integrating 3D segmentation with 2D CNN and RNN models. arXiv preprint arXiv:2407.16165.
Yao, Jiawei, & Jusheng Zhang. (2023). Depthssc: Depth-spatial alignment and dynamic voxel resolution for monocular 3d semantic scene completion. arXiv preprint arXiv:2311.17084.
Yao, Jiawei, et al. (2024). Building lane-level maps from aerial images. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE.
Zhang, X., Soe, A. N., Dong, S., Chen, M., Wu, M., & Htwe, T. (2024). Urban resilience through green roofing: A literature review on dual environmental benefits. in E3S Web of Conferences, 536, pp. 01023. EDP Sciences.
Ma, B., Ma, B., Gao, M., Wang, Z., Ban, X., Huang, H., & Wu, W. (2021). Deep learning‐based automatic inpainting for material microscopic images. Journal of Microscopy, 281(3), 177-189.
Dong, S., Xu, T., & Chen, M. (2022, October). Solar radiation characteristics in Shanghai. Journal of Physics: Conference Series, 2351(1), 012016). IOP Publishing.
Chen, M., Chen, Y., & Zhang, Q. (2024). Assessing global carbon sequestration and bioenergy potential from microalgae cultivation on marginal lands leveraging machine learning. Science of The Total Environment, 948, 174462.
Wang, Y., Ban, X., Wang, H., Li, X., Wang, Z., Wu, D., ... & Liu, S. (2019). Particle filter vehicles tracking by fusing multiple features. IEEE Access, 7, 133694-133706.
Zhu, Z., Wang, Z., Wu, Z., Zhang, Y., & Bo, S. (2024). Adversarial for sequential recommendation walking in the multi-latent space. Applied Science and Biotechnology Journal for Advanced Research, 3(4), 1-9.
Chen, M. (2023). Investigating the influence of interannual precipitation variability on terrestrial ecosystem productivity. Doctoral Dissertation, Massachusetts Institute of Technology.
Lu, Q., Guo, X., Yang, H., Wu, Z., & Mao, C. (2024). Research on adaptive algorithm recommendation system based on parallel data mining platform. Advances in Computer, Signals and Systems, 8(5), 23-33.
Wang, Randi, & Morad Behandish. (2022). Surrogate modeling for physical systems with preserved properties and adjustable tradeoffs. arXiv preprint arXiv:2202.01139.
Wang, Zixiang, et al. (2024). Research on autonomous robots navigation based on reinforcement learning. arXiv preprint arXiv:2407.02539.
Yan, Hao, et al. (2024). Research on image super-resolution reconstruction mechanism based on convolutional neural network. arXiv preprint arXiv:2407.13211.
Liu, Jiabei, et al. (2024). Application of deep learning-based natural language processing in multilingual sentiment analysis. Mediterranean Journal of Basic and Applied Sciences (MJBAS), 8(2), 243-260.
Xu, Qiming, et al. (2024). Applications of explainable AI in natural language processing. Global Academic Frontiers, 2(3), 51-64.
Zhong, Yihao, et al. (2024). Deep learning solutions for pneumonia detection: Performance comparison of custom and transfer learning models. medRxiv.
Tao Y. (2023). Meta learning enabled adversarial defense. IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE), pp. 1326-1330. IEEE.
Zhu, Armando, et al. (2024). Exploiting diffusion prior for out-of-distribution detection. arXiv preprint arXiv:2406.11105.
Li, Keqin, et al. (2024). Exploring the impact of quantum computing on machine learning performance.
Gu, Wenjun, et al. (2024). Predicting stock prices with FinBERT-LSTM: Integrating news sentiment analysis. arXiv preprint arXiv:2407.16150.
Wang, Zixiang, et al. (2024). Research on autonomous driving decision-making strategies based deep reinforcement learning. arXiv preprint arXiv:2408.03084.
Bo, Shi, et al. (2024). Attention mechanism and context modeling system for text mining machine translation. arXiv preprint arXiv:2408.04216.
Shimizu, Shosei et al. (2023). Boron neutron capture therapy for recurrent glioblastoma multiforme: Imaging evaluation of a case with long-term local control and survival. Cureus, 15(1), e33898. doi:10.7759/cureus.33898.
Bo, Shi, & Minheng Xiao. (2022). Dynamic risk measurement by EVT based on stochastic volatility models via MCMC. arXiv preprint arXiv:2201.09434.
Tao Y. (2023). SQBA: Sequential query-based blackbox attack. 5th International Conference on Artificial Intelligence and Computer Science (AICS 2023), 721-729.
Qian, Yang, et al. (2020). Heterogeneous optoelectronic characteristics of Si micropillar arrays fabricated by metal-assisted chemical etching. Scientific Reports, 10(1), 16349.
Li, Wei, et al. (2018). An intelligent electronic lock for remote-control system based on the internet of things. Journal of Physics: Conference Series, 1069(1). IOP Publishing.
Han, Yi, & Thomas CM Lee. (2022). Uncertainty quantification for sparse estimation of spectral lines. IEEE Transactions on Signal Processing 70, 6243-6256.
Han, Yi, & Thomas CM Lee. (2024). Structural break detection in non-stationary network vector autoregression models. IEEE Transactions on Network Science and Engineering.
Tan, Chaoyi, et al. (2024). Editable neural radiance fields convert 2D to 3D furniture texture. International Journal of Engineering and Management Research, 14(3), 62-65.
Xiao, Minheng, Shi Bo, & Zhizhong Wu. (2024). Multiple greedy quasi-newton methods for saddle point problems. arXiv preprint arXiv:2408.00241.
Niitsu, Hikaru et al. (2024). Tumor response on diagnostic imaging after proton beam therapy for hepatocellular carcinoma. Cancers, 16(2), 357. doi:10.3390/cancers16020357.
Pan, Xiaochao, et al. (2024). HarmonicNeRF: Geometry-informed synthetic view augmentation for 3D scene reconstruction in driving scenarios. ACM Multimedia.
Li, Zhenglin, et al. (2023). Stock market analysis and prediction using LSTM: A case study on technology stocks. Innovations in Applied Engineering and Technology, 1-6.
Mo, Yuhong, et al. (2024). Large Language Model (LLM) AI text generation detection based on transformer deep learning algorithm. International Journal of Engineering and Management Research, 14(2), 154-159.
Li, Shaojie, Yuhong Mo, & Zhenglin Li. (20220. Automated pneumonia detection in chest x-ray images using deep learning model. Innovations in Applied Engineering and Technology, 1-6.
Mo, Yuhong, et al. (2024). Password complexity prediction based on roberta algorithm. Applied Science and Engineering Journal for Advanced Research, 3(3), 1-5.
Song, Jintong, et al. (2024). A comprehensive evaluation and comparison of enhanced learning methods. Academic Journal of Science and Technology, 10(3), 167-171.
Liu, Tianrui, et al. (2024). Spam detection and classification based on distilbert deep learning algorithm. Applied Science and Engineering Journal for Advanced Research, 3(3), 6-10.
Dai, Shuying, et al. (2024). The cloud-based design of unmanned constant temperature food delivery trolley in the context of artificial intelligence. Journal of Computer Technology and Applied Mathematics, 1(1), 6-12.
Mo, Yuhong, et al. (2024). Make scale invariant feature transform “Fly” with CUDA. International Journal of Engineering and Management Research, 14(3), 38-45.
He, Shuyao, et al. (2024). Lidar and monocular sensor fusion depth estimation. Applied Science and Engineering Journal for Advanced Research, 3(3), 20-26.
Liu, Jihang, et al. (2024). Unraveling large language models: From evolution to ethical implications-introduction to large language models. World Scientific Research Journal, 10(5), 97-102.
Mo Yuhong, Zhang Yuchen, Li Hanzhe, Wang Han, & Yan Xu. (2024). Prediction of heart failure patients based on multiple machine learning algorithms. Applied and Computational Engineering, 75, 1-7. doi:10.54254/2755-2721/75/20240498.
Yan, H., Wang, Z., Bo, S., Zhao, Y., Zhang, Y., & Lyu, R. (2024). Research on image generation optimization based deep learning.
Tang, X., Wang, Z., Cai, X., Su, H., & Wei, C. (2024). Research on heterogeneous computation resource allocation based on data-driven method. arXiv preprint arXiv:2408.05671.
Wang, X. (2020). Nonlinear energy harvesting with tools from machine learning. Doctoral Dissertation, Duke University.
Qi, Z., Ma, D., Xu, J., Xiang, A., & Qu, H. (2024). Improved YOLOv5 based on attention mechanism and FasterNet for foreign object detection on railway and airway tracks. arXiv preprint arXiv:2403.08499.
Xiang, A., Huang, B., Guo, X., Yang, H., & Zheng, T. (2024). A neural matrix decomposition recommender system model based on the multimodal large language model. arXiv preprint arXiv:2407.08942.
Ma, D., Wang, M., Xiang, A., Qi, Z., & Yang, Q. (2024). Transformer-based classification outcome prediction for multimodal stroke treatment. arXiv preprint arXiv:2404.12634.
Xiang, A., Qi, Z., Wang, H., Yang, Q., & Ma, D. (2024). A multimodal fusion network for student emotion recognition based on transformer and tensor product. arXiv preprint arXiv:2403.08511.
Ma, D., Yang, Y., Tian, Q., Dang, B., Qi, Z., & Xiang, A. Comparative analysis of X-ray image classification of pneumonia based on deep learning algorithm algorithm.
Tan, C., Wang, C., Lin, Z., He, S., & Li, C. (2024). Editable neural radiance fields convert 2D to 3D furniture texture. International Journal of Engineering and Management Research, 14(3), 62-65
Wang, X. S., Turner, J. D., & Mann, B. P. (2021). Constrained attractor selection using deep reinforcement learning. Journal of Vibration and Control, 27(5-6), 502-514.
Kai Feng, Jingheng Wang, Xiaoyuan Wang, Gang Wang, Quanzheng Wang, & Junyan Han. (2024). Adaptive state estimation and filtering for dynamic positioning ships under time-varying environmental disturbances. Ocean Engineering.
Gang Wang, Jingheng Wang, Xiaoyuan Wang, Quanzheng Wang, Longfei Chen, Junyan Han, Bin Wang, & Kai Feng. (2024). Local path planning method for unmanned ship based on encounter situation inference and COLREGS constraints. Journal of Marine Science and Engineering.
Gang Wang, Jingheng Wang, Xiaoyuan Wang, Quanzheng Wang, Junyan Han, Longfei Chen, & Kai Feng. (2024). A method for coastal global route planning of unmanned ships based on human-like thinking. Journal of Marine Science and Engineering.
Bin Wang, Jingheng Wang, Xiaoyuan Wang, Longfei Chen, Han Zhang, Chenyang Jiao, Gang Wang, & Kai Feng. (2024). An identification method for road hypnosis based on human EEG data. Sensors (Basel).
Quanzheng Wang, Jingheng Wang, Xiaoyuan Wang, Luyao Wu, Kai Feng, & Gang Wang. (2024). A YOLOv7-based method for ship detection in videos of drones. Journal of Marine Science and Engineering.
Longfei Chen ,Jingheng Wang, Xiaoyuan Wang, Bin Wang, Han Zhang, Kai Feng, Gang Wang, Junyan Han, & Huili Shi. (2024). A road hypnosis identification method for drivers based on fusion of biological characteristics. Digital Transportation and Safety.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Zeyu Wang, Zong Cheng Chu, Minghao Chen, Yiqian Zhang, Rui Yang
This work is licensed under a Creative Commons Attribution 4.0 International License.
