Bug #5315: jenkins node rpi4 throws i/o errors, mounted root partition read-only - Cellular Network Infrastructure - Open Source Mobile Communications

Actions

Copy link

Bug #5315

closed

jenkins node rpi4 throws i/o errors, mounted root partition read-only

Added by osmith over 2 years ago. Updated over 2 years ago.

Status:

Resolved

Priority:

High

Assignee:

osmith

Target version:

Start date:

11/17/2021

Due date:

% Done:

100%

Spec Reference:

Description

After write errors, the SD card has been mounted read-only.

I guess somebody in the office should power off, dd the contents of this SD card to a new one, and boot again?

I'll mark the node in jenkins as temporary offline for now.

dmesg dmesg

[14762353.310716] blk_update_request: I/O error, dev mmcblk0, sector 34172976 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0                                                                                               
[14762353.310765] Buffer I/O error on dev mmcblk0p2, logical block 4205062, lost async page write                                                                                                                                 
[14762353.311434] Buffer I/O error on dev mmcblk0p2, logical block 4205063, lost async page write                                                                                                                                 
[14762353.312127] EXT4-fs error (device mmcblk0p2): ext4_check_bdev_write_error:216: comm rs:main Q:Reg: Error while async write back metadata                                                                                    
[14762353.312203] blk_update_request: I/O error, dev mmcblk0, sector 34173024 op 0x1:(WRITE) flags 0x103000 phys_seg 4 prio class 0                                                                                                                                             
[14762353.312238] Buffer I/O error on dev mmcblk0p2, logical block 4205068, lost async page write                                                                                                                                                                               
[14762353.312279] Buffer I/O error on dev mmcblk0p2, logical block 4205069, lost async page write                                                                                                                                                                               
[14762353.312316] Buffer I/O error on dev mmcblk0p2, logical block 4205070, lost async page write                                                                                                                                                                               
[14762353.312352] Buffer I/O error on dev mmcblk0p2, logical block 4205071, lost async page write                                                                                                                                                                               
[14762353.313686] blk_update_request: I/O error, dev mmcblk0, sector 34173080 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0                                                                                                                                             
[14762353.313719] Buffer I/O error on dev mmcblk0p2, logical block 4205075, lost async page write                                                                                                                                                                               
[14762353.314917] blk_update_request: I/O error, dev mmcblk0, sector 34173120 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0                                                                                                                                             
[14762353.314949] Buffer I/O error on dev mmcblk0p2, logical block 4205080, lost async page write                                                                                                                                                                               
[14762353.314988] Buffer I/O error on dev mmcblk0p2, logical block 4205081, lost async page write                                                                                                                                                                               
[14762353.316075] blk_update_request: I/O error, dev mmcblk0, sector 34173728 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0                                                                                                                                             
[14762353.316106] Buffer I/O error on dev mmcblk0p2, logical block 4205156, lost async page write                                                                                                                                                                               
[14762353.317330] blk_update_request: I/O error, dev mmcblk0, sector 34173744 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0                                                                                                                                             
[14762353.318572] blk_update_request: I/O error, dev mmcblk0, sector 34173904 op 0x1:(WRITE) flags 0x103000 phys_seg 5 prio class 0                                                                                                                                             
[14762353.319822] blk_update_request: I/O error, dev mmcblk0, sector 34174896 op 0x1:(WRITE) flags 0x103000 phys_seg 5 prio class 0                                                                                                                                             
[14762353.322015] blk_update_request: I/O error, dev mmcblk0, sector 34174944 op 0x1:(WRITE) flags 0x103000 phys_seg 12 prio class 0                                                                                                                                            
[14762353.323172] blk_update_request: I/O error, dev mmcblk0, sector 34175248 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0                                                                                                                                             
[14762353.644971] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061363 starting block 4324406)                                     
[14762353.645025] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061365 starting block 4324457)                                     
[14762353.645788] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061367 starting block 4324458)                                     
[14762353.645818] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061368 starting block 4324459)                                     
[14762353.646294] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061370 starting block 4324460)                                     
[14762353.646325] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061371 starting block 4324461)                                     
[14762353.646354] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061372 starting block 4324462)                                     
[14762353.646386] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061377 starting block 4324463)                                     
[14762353.646414] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061380 starting block 4324464)                                     
[14762353.646445] EXT4-fs warning (device mmcblk0p2): ext4_end_bio:349: I/O error 10 writing to inode 1061373 starting block 4324468)                                     
[14762353.646604] EXT4-fs error (device mmcblk0p2): ext4_check_bdev_write_error:216: comm kworker/u8:3: Error while async write back metadata                             
[14762353.717321] Aborting journal on device mmcblk0p2-8.                            
[14762353.717393] Buffer I/O error on device mmcblk0p2, logical block 606797                                                                                              
[14762353.718780] Buffer I/O error on device mmcblk0p2, logical block 606799                                                                                              
[14762353.720152] Buffer I/O error on device mmcblk0p2, logical block 2562736                                                                                             
[14762353.723258] Buffer I/O error on device mmcblk0p2, logical block 3340204                                                                                             
[14762353.780374] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected                                                                                          
[14762353.786283] JBD2: Error -5 detected when updating journal superblock for mmcblk0p2-8.                                                                               
[14762353.786927] EXT4-fs (mmcblk0p2): I/O error while writing superblock            
[14762353.786934] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected                                                                                          
[14762353.786951] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5749: Journal has aborted                                                                  
[14762353.789117] EXT4-fs (mmcblk0p2): I/O error while writing superblock            
[14762353.789133] EXT4-fs error (device mmcblk0p2): ext4_journal_check_start:83: Detected aborted journal                                                                 
[14762353.789168] EXT4-fs (mmcblk0p2): Remounting filesystem read-only               
[14762353.792028] EXT4-fs (mmcblk0p2): I/O error while writing superblock            
[14762353.792038] EXT4-fs error (device mmcblk0p2): __ext4_ext_dirty:169: inode #1061363: comm kworker/u8:3: mark_inode_dirty error                                       
[14762353.792082] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5749: Journal has aborted                                                                  
[14762353.799004] EXT4-fs (mmcblk0p2): I/O error while writing superblock            
[14762353.799043] EXT4-fs error (device mmcblk0p2): ext4_check_bdev_write_error:216: comm kworker/u8:3: Error while async write back metadata                             
[14762353.801662] EXT4-fs (mmcblk0p2): I/O error while writing superblock            
[14762353.801668] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected                                                                                          
[14762353.801687] EXT4-fs error (device mmcblk0p2) in ext4_reserve_inode_write:5749: Journal has aborted                                                                  
[14762353.804059] EXT4-fs (mmcblk0p2): I/O error while writing superblock            
[14762353.804087] EXT4-fs error (device mmcblk0p2): ext4_dirty_inode:5951: inode #1286: comm rs:main Q:Reg: mark_inode_dirty error                                        
[14762353.808689] EXT4-fs error (device mmcblk0p2): ext4_convert_unwritten_extents:4768: inode #1061363: comm kworker/u8:3: mark_inode_dirty error
[14762353.812005] Buffer I/O error on device mmcblk0p2, logical block 4257835
[14762353.812034] Buffer I/O error on device mmcblk0p2, logical block 4257836
[14762353.812055] Buffer I/O error on device mmcblk0p2, logical block 4257837
[14762353.812074] Buffer I/O error on device mmcblk0p2, logical block 4257838
[14762353.812094] Buffer I/O error on device mmcblk0p2, logical block 4257839
[14762353.812113] Buffer I/O error on device mmcblk0p2, logical block 4257840

Actions

Copy link

Updated by osmith over 2 years ago

powered it off via ssh, there's no use keeping it running if the SD card is damaged

Actions

Copy link

Updated by laforge over 2 years ago

Assignee changed from 4368 to roh

please let me know which size of uSD card we need to purchase. I'll try to get a high quality one.

Actions

Copy link

Updated by roh over 2 years ago

Status changed from New to In Progress

refitted a new sdcard, installed recent raspios buster (2021-05-07-raspios-buster-armhf-lite.img)

i tried giving it the correct ip via /etc/network/interfaces but that fails due to bridge-utils or sth missing... i guess the ansible pulls that in? not sure...

its accessible as 10.9.25.132 for now with 'sysmocom23' as password for user pi.

the old sdcard was a sandisk extreme pro 64gb
the new one is a samsung evo 64gb - should be ok.

osmith can you take over from here?

Actions

Copy link