Project

General

Profile

Actions
Copy link

Bug #4641

closed

osmo-ggsn: heap-use-after-free in sgsn_peer_drop_all_pdp_except

Added by pespin almost 4 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
pespin
Category:
-
Target version:
-
Start date:
07/03/2020
Due date:
% Done:

100%

Spec Reference:

Description

Got it a few seconds after killing (restarting) osmo-sgsn:

20200703132046337 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132046348 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132047218 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132047230 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132048335 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132048346 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132049416 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132049427 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132050417 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132050429 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132051335 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132051347 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132051636 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132051636 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132052318 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132052330 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132053256 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:653 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20200703132053268 DTUN <0001> /git/osmo-ggsn/ggsn/ggsn.c:632 TUN(tun4): Received packet for APN(internet)
20200703132151636 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132151637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132251637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132251637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132351637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132351637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132451637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132451637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132551637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132551637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132651637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132651638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132751637 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132751638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:151 SGSN(127.0.0.1): SGSN recovery (174->175) pdp=(nil), releasing all PDP contexts
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:66 PDP(901700000015256:5): Sending DELETE PDP CTX due to shutdown
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/ggsn.c:354 PDP(901700000015256:5): Deleting PDP context
20200703132851638 DGGSN <0002> /git/osmo-ggsn/ggsn/sgsn.c:21 SGSN(127.0.0.1): Deleting SGSN
20200703132851638 DLGTP <000d> /git/osmo-ggsn/gtp/pdp.c:296 Begin pdp_tiddel tid = 5652510000007109
20200703132851638 DLGTP <000d> /git/osmo-ggsn/gtp/pdp.c:303 End pdp_tiddel: PDP found
=================================================================
==12028==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000005848 at pc 0x5555555da90f bp 0x7fffffffb7f0 sp 0x7fffffffb7e0
READ of size 8 at 0x611000005848 thread T0
    #0 0x5555555da90e in sgsn_peer_drop_all_pdp_except /git/osmo-ggsn/ggsn/sgsn.c:123
    #1 0x5555555db031 in sgsn_peer_handle_recovery /git/osmo-ggsn/ggsn/sgsn.c:157
    #2 0x5555555d6a05 in cb_recovery3 /git/osmo-ggsn/ggsn/ggsn.c:782
    #3 0x7ffff74fc66b in emit_cb_recovery /git/osmo-ggsn/gtp/gtp.c:223
    #4 0x7ffff7508769 in gtp_echo_conf /git/osmo-ggsn/gtp/gtp.c:1134
    #5 0x7ffff752a9e1 in gtp_decaps1c /git/osmo-ggsn/gtp/gtp.c:3154
    #6 0x5555555d661e in ggsn_gtp_fd_cb /git/osmo-ggsn/ggsn/ggsn.c:725
    #7 0x7ffff699ef76 in osmo_fd_disp_fds /git/libosmocore/src/select.c:227
    #8 0x7ffff699f35b in _osmo_select_main /git/libosmocore/src/select.c:265
    #9 0x7ffff699f43a in osmo_select_main /git/libosmocore/src/select.c:274
    #10 0x5555555bb31c in main /git/osmo-ggsn/ggsn/ggsn_main.c:201
    #11 0x7ffff5d3a001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)
    #12 0x5555555bab0d in _start (/build/new/out/bin/osmo-ggsn+0x66b0d)

0x611000005848 is located 136 bytes inside of 240-byte region [0x6110000057c0,0x6110000058b0)
freed by thread T0 here:
    #0 0x7ffff766b0e9 in __interceptor_free /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:123
    #1 0x7ffff689941b  (/usr/lib/libtalloc.so.2+0x441b)

previously allocated by thread T0 here:
    #0 0x7ffff766b459 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x7ffff689bb8c  (/usr/lib/libtalloc.so.2+0x6b8c)

SUMMARY: AddressSanitizer: heap-use-after-free /git/osmo-ggsn/ggsn/sgsn.c:123 in sgsn_peer_drop_all_pdp_except
Shadow bytes around the buggy address:
  0x0c227fff8ab0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8ac0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8ad0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8ae0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
  0x0c227fff8af0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c227fff8b00: fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd
  0x0c227fff8b10: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8b20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==12028==ABORTING
[Inferior 1 (process 12028) exited with code 01]
(gdb)
Actions
Copy link
 #1

Updated by pespin almost 4 years ago

Using osmo-ggsn.git 4e37fb356aafda0b12d8b33daa5057c43fe633f5

Failure line is:

llist_for_each_entry_safe(pdp, pdp2, &sgsn->pdp_list, entry) {

So it looks like some pdp context is left in the pdp_list after being freed (probably by libgtp?).

Actions
Copy link
 #2

Updated by pespin almost 3 years ago

I got this today, potentially after restarting SGSN:
osmo-ggsn.git bd2b55679e897b8f2ef14bf24e4e17967098c03f

20210602190922315 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190922352 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:681 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20210602190922363 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190922365 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:681 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20210602190922376 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190922404 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:681 PDP(901700000015256:5): Packet received on APN(internet): forwarding to tun tun4
20210602190922415 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 8.8.8.8
20210602190923042 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190925090 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190925490 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190929122 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190932390 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190933810 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190937378 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190950194 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190953763 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602190956947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602190956947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191002599 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602191032807 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602191056947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191056947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191101364 DTUN <0001> /osmo-ggsn/ggsn/ggsn.c:643 TUN(tun4): APN(internet) Rx DL data packet for PDP(901700000015256:5): 176.16.222.5 <- 142.250.200.132
20210602191156947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191156947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191256947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191256947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191356947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191356947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191456947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191456947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191556947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191556947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191656947 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:29 SGSN(127.0.0.1): Tx Echo Request
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:40 SGSN(127.0.0.1): Rx Echo Response
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:151 SGSN(127.0.0.1): SGSN recovery (62->63) pdp=(nil), releasing all PDP contexts
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:66 PDP(901700000015256:5): Sending DELETE PDP CTX due to shutdown
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/ggsn.c:354 PDP(901700000015256:5): Deleting PDP context
20210602191656948 DGGSN <0002> /osmo-ggsn/ggsn/sgsn.c:21 SGSN(127.0.0.1): Deleting SGSN
20210602191656948 DLGTP <000d> /osmo-ggsn/gtp/pdp.c:296 Begin pdp_tiddel tid = 5652510000007109
20210602191656948 DLGTP <000d> /osmo-ggsn/gtp/pdp.c:303 End pdp_tiddel: PDP found
=================================================================
==221315==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000006388 at pc 0x5555555df532 bp 0x7fffffffbaa0 sp 0x7fffffffba90
READ of size 8 at 0x611000006388 thread T0
    #0 0x5555555df531 in sgsn_peer_drop_all_pdp_except /osmo-ggsn/ggsn/sgsn.c:123
    #1 0x5555555dfc9e in sgsn_peer_handle_recovery /osmo-ggsn/ggsn/sgsn.c:157
    #2 0x5555555db31d in cb_recovery3 /osmo-ggsn/ggsn/ggsn.c:810
    #3 0x7ffff75007ab in emit_cb_recovery /osmo-ggsn/gtp/gtp.c:230
    #4 0x7ffff750cf9a in gtp_echo_conf /osmo-ggsn/gtp/gtp.c:1141
    #5 0x7ffff7530d83 in gtp_decaps1c /osmo-ggsn/gtp/gtp.c:3228
    #6 0x5555555daf09 in ggsn_gtp_fd_cb /osmo-ggsn/ggsn/ggsn.c:753
    #7 0x7ffff68299a1 in poll_disp_fds /libosmocore/src/select.c:350
    #8 0x7ffff6829af6 in _osmo_select_main /libosmocore/src/select.c:378
    #9 0x7ffff6829b15 in osmo_select_main /libosmocore/src/select.c:417
    #10 0x5555555bd74e in main /osmo-ggsn/ggsn/ggsn_main.c:249
    #11 0x7ffff5ba7b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
    #12 0x5555555bcb3d in _start (/home/pespin/dev/sysmocom/build/new/out/bin/osmo-ggsn+0x68b3d)

20210602183719783 DGGSN <0002> /osmo-ggsn/ggsn/pco.c:205 PDP(901700000015256:5): PCO Protocol 0xc223
0x611000006388 is located 136 bytes inside of 240-byte region [0x611000006300,0x6110000063f0)
freed by thread T0 here:
    #0 0x7ffff7676f19 in __interceptor_free /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:127
    #1 0x7ffff671141b  (/usr/lib/libtalloc.so.2+0x441b)

previously allocated by thread T0 here:
    #0 0x7ffff7677279 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x7ffff6713b8c  (/usr/lib/libtalloc.so.2+0x6b8c)

SUMMARY: AddressSanitizer: heap-use-after-free /osmo-ggsn/ggsn/sgsn.c:123 in sgsn_peer_drop_all_pdp_except
Shadow bytes around the buggy address:
  0x0c227fff8c20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
  0x0c227fff8c30: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8c40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8c50: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8c60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c227fff8c70: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fa fa
  0x0c227fff8c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8cc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==221315==ABORTING
[Inferior 1 (process 221315) exited with code 01]
Actions
Copy link
 #3

Updated by pespin almost 3 years ago

  • Assignee set to pespin
Actions
Copy link
 #4

Updated by pespin almost 3 years ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 90

Should be fixed by:
https://gerrit.osmocom.org/c/osmo-ggsn/+/24642 ggsn: Fix heap-use-after-free during Recovery without associated PDP

Ticket can be closed when it gets merged.

Actions
Copy link
 #5

Updated by pespin almost 3 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 90 to 100

Merged, closing.

Actions
Copy link

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)