path: root/results/scraper/launchpad/1994002

[SRU] migration was active, but no RAM info was set

While live-migrating many instances concurrently, libvirt sometimes return internal error: migration was active, but no RAM info was set:
~~~
2022-03-30 06:08:37.197 7 WARNING nova.virt.libvirt.driver [req-5c3296cf-88ee-4af6-ae6a-ddba99935e23 - - - - -] [instance: af339c99-1182-4489-b15c-21e52f50f724] Error monitoring migration: internal error: migration was active, but no RAM info was set: libvirt.libvirtError: internal error: migration was active, but no RAM info was set
~~~

From upstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=2074205

[Impact]

 * Effects of this bug are mostly observed in large scale clusters with a lot of live migration activity.
 * Has second order effects for consumers of migration monitor such as libvirt and openstack.

[Test Case]
Steps to Reproduce:
1. live evacuate a compute
2. live migration of one or more instances fails with the above error

N.B Due to the nature of this bug it is difficult consistently reproduce.

[Where problems could occur]
 * In the event of a regression the migration monitor may report an inconsistent state.



The attachment "lp1994002-qemu-ussuri.debdiff" seems to be a debdiff.  The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff.  If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

If you need something from upstream QEMU, please use the new bug tracker here: https://gitlab.com/qemu-project/qemu/-/issues







Hi Brett,

Thanks for the debdiffs!

I just reviewed them, and there are changes that should be made.

I could do those myself, but that wouldn't be an opportunity to
learn/practice some details for SRUs for you, so I'll add notes.

*However*, if you're too busy and can't do that, do let me know.

cheers,
Mauricio

...

qemu.git

$ git describe --contains 552de79bfdd5e9e53847eb3c6d6e4cd898a4370e
v7.1.0-rc0~136^2

ubuntu archive:

$ rmadison -a source qemu
...
 qemu | 1:2.11+dfsg-1ubuntu7    | bionic          | source
 qemu | 1:2.11+dfsg-1ubuntu7.40 | bionic-security | source
 qemu | 1:2.11+dfsg-1ubuntu7.40 | bionic-updates  | source
 qemu | 1:4.2-3ubuntu6          | focal           | source
 qemu | 1:4.2-3ubuntu6.23       | focal-security  | source
 qemu | 1:4.2-3ubuntu6.23       | focal-updates   | source
 qemu | 1:6.2+dfsg-2ubuntu6     | jammy           | source
 qemu | 1:6.2+dfsg-2ubuntu6.2   | jammy-security  | source
 qemu | 1:6.2+dfsg-2ubuntu6.5   | jammy-updates   | source
 qemu | 1:7.0+dfsg-7ubuntu2     | kinetic         | source
 qemu | 1:7.0+dfsg-7ubuntu2     | lunar           | source

0) Development release

The development release (lunar) still doesn't have the patch.
That is required for SRU / stable releases.

We'll need a debdiff for lunar, slightly different than kinetic
(release name and greater version string for the upgrade path).

I just checked w/ Christian and we shouldn't wait on qemu 7.1
merge from Debian (sid), which would include the patch, since
the merge from Debian should happen in January to get qemu 7.2.


1) Oldest LTS in standard support

Would Bionic benefit from this fix on the long run as well,
just before it goes into expanded/out of standard- support?

Apparently, some deployments/clouds still use Bionic on kvm
compute nodes.

If so, the backport targets qmp_query_migrate()/same file,
per commit 65ace0604551 ("migration: add postcopy total blocktime into query-migrate").


2) Debdiffs:

- version strings: the 'lp*' version suffix is fine for
test builds, but for official packages usually (see [1]):
just increment '.1' on stable releases, and '1' on dev.

example:
kinetic (sru): 1:7.0+dfsg-7ubuntu2 -> ubuntu2.1
luanr (devel): 1:7.0+dfsg-7ubuntu2 -> ubuntu3

- changelog: mostly good! (d/p/file.patch; LP: #number?; releases).

The LP bug number 1982284 refers to another/openstack bug,
but the Ubuntu SRUs are coming through this bug, apparently.

Since this is the bug where Ubuntu Archive/Cloud Archive
have packages/series on, to be closed when SRUs land in
-proposed and -updates (and UCA), we should change:
1) the LP bug number in the changelog 
2) and patch file names
3) also, it's a good idea to link to other LP bug
in the SRU template '[Other Info]' section.

(you could also just move the SRU template/packages/
series/tracks to the other LP bug, I guess. Up to you.)

- quilt patch: add DEP3 headers [2] (Origin:/Bug-Ubuntu:)

- quilt series: missing 'ubuntu/' dir on k/j (not on f)

- duplications: jammy has duplicated messages, and focal
has that plus duplicated changelog entries? -- for HA? x)

[1] https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging
[2] https://dep-team.pages.debian.net/deps/dep3/

Brett, per our email conversation, please ignore this:
> - quilt series: missing 'ubuntu/' dir on k/j (not on f)

I missed that focal uses `d/p/ubuntu/` too (it just wasn't present in `d/p/series` context lines in the debdiff, for CVEs).
Sorry for the confusion, and thanks for checking!


Hi Mauricio,

Thanks for your review.
I've made the changes you've requested.
Looking forward to your feedback.










Thanks, Brett!

Very minor nitpicks left (changelog entry/release for lunar,
and URL for Origin:), I can handle those.

For Lunar/devel release, I'll send a MR for Christian to
review/upload (my upload rights are for stable releases).

This includes a fix to FTBFS per a package change in the
last 24 hours :) happy to catch it now!

It's currently (re)build-testing on all supported archs.
If all goes well now, I'll send the MR for Lunar, and
once it lands, we'll proceed w/ SRUs.

...

I also played with GDB for a synthetic reproducer.
It seems to be possible, but needs a little more study
on the monitor path.  We can sync on that later!


All archs finished building successfully on ppa:mfo/lp1994002v2.

Just sent the MR for Lunar. If/once it lands, I can do the SRUs.

https://code.launchpad.net/~mfo/ubuntu/+source/qemu/+git/qemu/+merge/434118

For documentation purposes,

The qemu package in lunar-proposed has its migration blocked to lunar(-release)
because of autopkgtests failures (sbuild), which have been analyzed/understood.

We're waiting on the autopkgtests queue to run sbuild w/ triggers on qemu _and_
sbuild from lunar-proposed, which should address the error w/ sbuild/unshare
(lack of adduser command in the sbuild chroot, as apt no longer deps on that).

Once that runs, we'll check if any other errors happen, and address those.

cheers,
Mauricio

The sbuild autopkgtest failure on the 'unshare' test
is indeed fixed w/ sbuild in lunar-proposed; however,
now the test 'unshare-qemuwrapper' timed out.

autopkgtest [23:36:43]: @@@@@@@@@@@@@@@@@@@@ summary
build-procenv        PASS
unshare-qemuwrapper  FAIL timed out
unshare              PASS

It timed out on the 'guestfish' command, so I enabled
`export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1` there,
and run autopkgtests against its build in a PPA [1].

Then it finished successfully w/out timing out! x)

autopkgtest [16:17:39]: @@@@@@@@@@@@@@@@@@@@ summary
build-procenv        PASS
unshare-qemuwrapper  PASS
unshare              PASS

Not a very useful result, but it did show that an
step in guestfish took ~25 minutes; 30 mins total:

autopkgtest [15:22:52]: test unshare-qemuwrapper: [-----------------------
...
+ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
+ guestfish <...>
...
libguestfs: trace: tar_in "/tmp/.../ubuntu-lunar-host.tar" "/"
...
tar -C /sysroot/ -xf - 2> /tmp/tarSfYHJX
...
guestfsd: => tar_in (0x45) took 1489.08 secs
...
autopkgtest [15:52:27]: test unshare-qemuwrapper: -----------------------]
unshare-qemuwrapper  PASS

So, well, it might have been due to load in the
autopkgtest infrastructure at the time tests ran,
so just triggered retries on sbuild and sbuild+qemu.

Hopefully they will pass and unblock proposed migration
for both sbuild & qemu.

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar-mfo-build/lunar/amd64/s/sbuild/20221215_161801_a2772@/log.gz



The sbuild autopkgtests need a fix for lunar-proposed;
reported bug 2000015 w/ analysis and debdiff attached.

This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu3

---------------
qemu (1:7.0+dfsg-7ubuntu3) lunar; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/ebpf-replace-deprecated-bpf_program__set_socket_filt.patch:
    Fix FTBFS with libbpf 1.0.1-2.

 -- Mauricio Faria de Oliveira <email address hidden>  Wed, 30 Nov 2022 12:17:51 -0300

These are the steps for a synthetic reproducer with GDB and QEMU.

It's sufficient to validate the change on QEMU and move forward with SRUs.
(We can add libvirt for reaching the error message too, but it's a plus.)

I'll check/coordinate on the QEMU uploads next week.

cheers,
Mauricio

...


Original code.

The race condition is, one thread can read 's->state' as SETUP (line 1078/1083/1086),
and another thread changes it to ACTIVE _before_ this thread assigns (the different)
's->state' to 'info->status' (line 1123), which will NOT have RAM info expected for
the ACTIVE status (line 1087/1096).

	1056 static void fill_source_migration_info(MigrationInfo *info)
	1057 {
	1058     MigrationState *s = migrate_get_current();
	...
	1078     switch (s->state) {
	...
	1083     case MIGRATION_STATUS_SETUP:
	...
	1086         break;
	1087     case MIGRATION_STATUS_ACTIVE:
	...
	1096         populate_ram_info(info, s);
	...
	1123     info->status = s->state;
	1124 }

We'll break on this function that changes migration states.

	void migrate_set_state(int *state, int old_state, int new_state)


Terminal 1)

QEMU process for incoming migration (ie, destination)

qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

QEMU process for outgoing migration (ie, source)
with QEMU monitor on port 3333,
with GDB in non-stop threads mode
(and debuginfod for debug symbols),


gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  -iex 'set debuginfod enabled on' -iex 'set debuginfod urls "https://debuginfod.ubuntu.com"' \
  qemu-system-x86_64

Set breakpoints on the functions above
(and line number of the 'break' statement under 'case MIGRATION_STATUS_SETUP'):

b migrate_set_state
b migration/migration.c:1086

run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off
...

Terminal 3)

Connect to QEMU monitor with netcat,
and start the the migration:

nc 127.0.0.1 3333
...
(qemu) migrate -d tcp:127.0.0.1:4444


Terminal 2)

GDB breaks on change from migration status NONE to SETUP. 

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=0, new_state=1) at ../../migration/migration.c:1746

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE

(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP

Just continue.

(gdb) c

GDB breaks on change from migration status SETUP to ACTIVE. 

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=1, new_state=4) at ../../migration/migration.c:1746

(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP

(gdb) p (MigrationStatus) 4
$3 = MIGRATION_STATUS_ACTIVE

Let's explore the race condition.


Terminal 2)

Check the migration information:

(qemu) info migrate


Terminal 3)

GDB breaks on the migration information function.

The status is now observed as SETUP (not yet ACTIVE),
and is not yet propagated to the migration information
to be returned to the monitor.

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556d65c70) at ../../migration/migration.c:1086

(gdb) p (MigrationStatus) s.state
$4 = MIGRATION_STATUS_SETUP

(gdb) p info.status
$5 = MIGRATION_STATUS_NONE


Now, allow the other thread to continue and change status to ACTIVE.

(gdb) info threads
  Id   Target Id                                          Frame
* 1    Thread 0x7ffff6c21e80 (LWP 995) "qemu-system-x86"  fill_source_migration_info (info=0x555556d65c70) at ../../migration/migration.c:1123
  2    Thread 0x7ffff65ff640 (LWP 998) "qemu-system-x86"  (running)
  3    Thread 0x7ffff5d7c640 (LWP 999) "qemu-system-x86"  (running)
  5    Thread 0x7ffff49ff640 (LWP 1002) "qemu-system-x86" migrate_set_state (state=0x5555566f8628, old_state=1, new_state=4) at ../../migration/migration.c:1746

(gdb) thread 5
[Switching to thread 5 (Thread 0x7ffff49ff640 (LWP 1002))]
#0  migrate_set_state (state=0x5555566f8628, old_state=1, new_state=4) at ../../migration/migration.c:1746
1746    in ../../migration/migration.c

(gdb) continue &
Continuing.

The first thread now observes the ACTIVE status.

(gdb) info threads
  Id   Target Id                                          Frame
  1    Thread 0x7ffff6c21e80 (LWP 995) "qemu-system-x86"  fill_source_migration_info (info=0x555556d65c70) at ../../migration/migration.c:1123
  2    Thread 0x7ffff65ff640 (LWP 998) "qemu-system-x86"  (running)
  3    Thread 0x7ffff5d7c640 (LWP 999) "qemu-system-x86"  (running)
* 5    Thread 0x7ffff49ff640 (LWP 1002) "qemu-system-x86" (running)
(gdb)

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff6c21e80 (LWP 995))]
#0  fill_source_migration_info (info=0x555556d65c70) at ../../migration/migration.c:1123
1123    in ../../migration/migration.c

(gdb) p (MigrationStatus) s.state
$7 = MIGRATION_STATUS_ACTIVE


Done, the next statement in the code assigns the ACTIVE
status to the migration information, but it did NOT add
any RAM statistics (as it executed for the SETUP status).

Just continue, and check the resulting migration info:

(gdb) c


Terminal 2)
... info migrate
...
Migration status: active
total time: 0 ms
(qemu)

Migration status is active, without any RAM statistics.


...
...
...

Note that, normally, without exploring the race condition,
the status is 'active' and there _is_ memory statistics:

[just a summary from gdb/monitor steps]

...
(qemu) migrate -d tcp:127.0.0.1:4444

...
Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=0, new_state=1) at ../../migration/migration.c:1746
(gdb) c
Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=1, new_state=4) at ../../migration/migration.c:1746
(gdb) thread 5
(gdb) c
Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=4, new_state=8) at ../../migration/migration.c:1746

(qemu) info migrate
...
Migration status: active
total time: 41387 ms
expected downtime: 300 ms
setup: 22590 ms
transferred ram: 485 kbytes
throughput: 0.18 mbps
remaining ram: 0 kbytes
total ram: 131592 kbytes
duplicate: 32849 pages
skipped: 0 pages
normal: 49 pages
normal bytes: 196 kbytes
dirty sync count: 3
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 1453
(qemu)

(gdb) c
...


Uploaded to k/j/f/b.

Hello Brett, or anyone else affected,

Accepted qemu into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:7.0+dfsg-7ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.  Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted qemu (1:7.0+dfsg-7ubuntu2.2) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

sbuild/0.83.1ubuntu1 (amd64)
ubuntu-image/2.2+22.04ubuntu3 (ppc64el)
dropbear/2022.82-4 (armhf)


Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!


Autopkgtests regressions cleared for kinetic.

Verification done on kinetic-proposed.

The migration status during the race condition is
not 'active' (which is expected to have RAM info, but didn't)
but 'setup' (which is not, thus no issues).

(qemu) info migrate

-updates:
...
Migration status: active
total time: 0 ms

-proposed:
...
Migration status: setup
total time: 0 ms

Detailed steps:
==============

$ lxc launch ubuntu:kinetic qemu-k
$ lxc exec qemu-k -- su - ubuntu


Packages from -updates: FAIL
----------------------

$ sudo apt install --yes --no-install-recommends qemu-system-x86 gdb dpkg-dev

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:7.0+dfsg-7ubuntu2.1

...

Source: get line number for breakpoint.

$ sudo add-apt-repository -ys
$ apt source qemu

$ head -n1 qemu-*/debian/changelog
qemu (1:7.0+dfsg-7ubuntu2.1) kinetic-security; urgency=medium

$ vim qemu-*/migration/migration.c

1073 static void fill_source_migration_info(MigrationInfo *info)
1074 {
...
1100     case MIGRATION_STATUS_SETUP:
...
1103         break;
...

...

Terminal 1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  -iex 'set debuginfod enabled on' -iex 'set debuginfod urls https://debuginfod.ubuntu.com' \
  qemu-system-x86_64
  
(gdb) b migrate_set_state
...
Breakpoint 1 at 0x47ed10: migrate_set_state. (2 locations)
(gdb) b migration/migration.c:1103
...
Breakpoint 2 at 0x47dba0: file ../../migration/migration.c, line 1103.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off


Terminal 3)

nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x555556779618, old_state=0, new_state=1) at ../../migration/migration.c:1763

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x555556779618, old_state=1, new_state=4) at ../../migration/migration.c:1763

(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

Terminal 3)

(qemu) info migrate

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1103

(gdb) p (MigrationStatus) s.state
$6 = MIGRATION_STATUS_SETUP
(gdb) p info.status
$7 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id   Target Id                                          Frame
* 1    Thread 0x7ffff6c32340 (LWP 2368) "qemu-system-x86" fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1103
  2    Thread 0x7ffff65ff6c0 (LWP 2369) "qemu-system-x86" (running)
  3    Thread 0x7ffff5d7c6c0 (LWP 2370) "qemu-system-x86" (running)
  5    Thread 0x7ffff49ff6c0 (LWP 2373) "qemu-system-x86" migrate_set_state (state=0x555556779618, old_state=1, new_state=4) at ../../migration/migration.c:1763

(gdb) thread 5
(gdb) continue &

(gdb) info threads
  Id   Target Id                                          Frame
  1    Thread 0x7ffff6c32340 (LWP 2368) "qemu-system-x86" fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1103
  2    Thread 0x7ffff65ff6c0 (LWP 2369) "qemu-system-x86" (running)
  3    Thread 0x7ffff5d7c6c0 (LWP 2370) "qemu-system-x86" (running)
* 5    Thread 0x7ffff49ff6c0 (LWP 2373) "qemu-system-x86" (running)

(gdb) thread 1

(gdb) p (MigrationStatus) s.state
$8 = MIGRATION_STATUS_ACTIVE
(gdb) c

Terminal 3)

...
Migration status: active
total time: 0 ms
(qemu)

Migration status is active, without any RAM statistics.

(qemu) quit
(gdb) quit

Terminal 1)

Ctrl-C

...


Packages from -proposed: PASS
-----------------------

$ sudo add-apt-repository -yp proposed
$ sudo add-apt-repository -ys # didn't work for proposed
$ echo 'deb-src http://archive.ubuntu.com/ubuntu kinetic-proposed main' | sudo tee -a /etc/apt/sources.list

$ sudo apt install --yes --no-install-recommends qemu-system-x86

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:7.0+dfsg-7ubuntu2.2

$ rm -rf qemu-*
a$ apt source qemu

$ head -n1 qemu-*/debian/changelog
qemu (1:7.0+dfsg-7ubuntu2.2) kinetic; urgency=medium

$ vim qemu-*/migration/migration.c

1073 static void fill_source_migration_info(MigrationInfo *info)
1074 {
...
1076     int state = qatomic_read(&s->state);
...
1101     case MIGRATION_STATUS_SETUP:
...
1104         break;


Terminal 1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

$ gdb   -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off'   -iex 'set debuginfod enabled on' -iex 'set debuginfod urls https://debuginfod.ubuntu.com'   qemu-system-x86_64

(gdb) b migrate_set_state
...
Breakpoint 1 at 0x47ed20: migrate_set_state. (2 locations)
(gdb) b migration/migration.c:1104
...
Breakpoint 2 at 0x47dbc3: file ../../migration/migration.c, line 1104.

Terminal 3)

$ nc 127.0.0.1 3333
(qemu) migrate -d tcp:127.0.0.1:4444

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x555556779618, old_state=0, new_state=1) at ../../migration/migration.c:1764

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x555556779618, old_state=1, new_state=4) at ../../migration/migration.c:1764
1764    in ../../migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

(qemu) info migrate

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1141

(gdb) p (MigrationStatus) s.state
$6 = MIGRATION_STATUS_SETUP
(gdb) p info.status
$7 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id   Target Id                                          Frame
* 1    Thread 0x7ffff6c32340 (LWP 7562) "qemu-system-x86" fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1141
  2    Thread 0x7ffff65ff6c0 (LWP 7565) "qemu-system-x86" (running)
  3    Thread 0x7ffff5d7c6c0 (LWP 7566) "qemu-system-x86" (running)
  5    Thread 0x7fffa7dff6c0 (LWP 7569) "qemu-system-x86" migrate_set_state (state=0x555556779618, old_state=1, new_state=4) at ../../migration/migration.c:1764

(gdb) thread 5
(gdb) continue &

(gdb) info threads
  Id   Target Id                                          Frame
  1    Thread 0x7ffff6c32340 (LWP 7562) "qemu-system-x86" fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1141
  2    Thread 0x7ffff65ff6c0 (LWP 7565) "qemu-system-x86" (running)
  3    Thread 0x7ffff5d7c6c0 (LWP 7566) "qemu-system-x86" (running)
* 5    Thread 0x7fffa7dff6c0 (LWP 7569) "qemu-system-x86" (running)

(gdb) thread 1
(gdb) p (MigrationStatus) s.state
$8 = MIGRATION_STATUS_ACTIVE

(gdb) c

Terminal 3)

Status is now still 'SETUP' (which is not expected to have RAM statistics), not 'ACTIVE' (which is, and caused the issue).

...
Migration status: setup
total time: 0 ms


This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu2.2

---------------
qemu (1:7.0+dfsg-7ubuntu2.2) kinetic; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)

 -- Mauricio Faria de Oliveira <email address hidden>  Thu, 02 Mar 2023 17:29:05 -0300

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates.  Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report.  In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Brett, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.  Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Brett, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.  Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Brett, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.42 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.  Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.7) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)
ubuntu-image/2.2+22.04ubuntu3 (amd64, arm64, ppc64el)


Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!


All autopkgtests for the newly accepted qemu (1:4.2-3ubuntu6.25) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.664.46 (amd64)


Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!


All autopkgtests for the newly accepted qemu (1:2.11+dfsg-1ubuntu7.42) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/237-3ubuntu10.57 (i386)


Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!


autopkgtests regressions cleared; unrelated to these changes.

Verification done on jammy-proposed.

Followed the instructions as per: https://bugs.launchpad.net/cloud-archive/+bug/1994002/comments/26

With the exception that I had to install the debug symbols package as per https://wiki.ubuntu.com/Debug%20Symbol%20Packages for -proposed.

jammy-updates -- Fail:
ubuntu@qemu-j:~$ nc 127.0.0.1 3333
QEMU 6.2.0 monitor - type 'help' for more information
(qemu) migrate -d tcp:127.0.0.1:4444
migrate -d tcp:127.0.0.1:4444

(qemu)
(qemu) info migrate
info migrate

globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 0 ms
(qemu)
(qemu) quit
quit

jammy-proposed - Pass:
ubuntu@qemu-j2:~$ nc 127.0.0.1 3333
QEMU 6.2.0 monitor - type 'help' for more information
(qemu) migrate -d tcp:127.0.0.1:4444
migrate -d tcp:127.0.0.1:4444
(qemu) info migrate
info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: setup
total time: 0 ms
(qemu)

Full gdb session output: https://pastebin.ubuntu.com/p/mkhQzCXKdk/

Verification done for focal-proposed.

focal-updates: FAIL (status: active)

	(qemu) info migrate
	...
	Migration status: active
	total time: 0 milliseconds

focal-proposed: PASS (status: setup)

	(qemu) info migrate
	...
	Migration status: setup
	total time: 0 milliseconds

Details:
=======

$ lsb_release -cs
focal

focal-updates: FAIL
-------------

$ curl http://ddebs.ubuntu.com/dbgsym-release-key.asc | sudo apt-key add -
$ sudo add-apt-repository -y 'deb http://ddebs.ubuntu.com/ubuntu focal-updates main'

$ sudo apt install --yes qemu-system-x86 qemu-system-x86-dbgsym

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:4.2-3ubuntu6.24

$ dpkg -s qemu-system-x86-dbgsym | grep Version:
Version: 1:4.2-3ubuntu6.24

...

$ sudo add-apt-repository -ys 'deb http://archive.ubuntu.com/ubuntu focal-updates main'

$ apt source qemu
$ head -n1 qemu-*/debian/changelog
qemu (1:4.2-3ubuntu6.24) focal-security; urgency=medium

$ vim qemu-*/migration/migration.c

 915 static void fill_source_migration_info(MigrationInfo *info)
...
 925     case MIGRATION_STATUS_SETUP:
 926         info->has_status = true;
 927         info->has_total_time = false;
 928         break;

...

T1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

T2)

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  qemu-system-x86_64
  
(gdb) b migrate_set_state
Breakpoint 1 at 0x6d3aa0: migrate_set_state. (2 locations)

(gdb) b migration/migration.c:928
Breakpoint 2 at 0x6d317b: file ./migration/migration.c, line 928.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off

T3)

nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566949d8, old_state=0, new_state=1) at ./migration/migration.c:1463
1463    ./migration/migration.c: No such file or directory.
(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1463
1463    in ./migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE
(gdb)

T3)

(qemu) info migrate

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
928     in ./migration/migration.c

(gdb) p (MigrationStatus) s.state
$6 = MIGRATION_STATUS_SETUP
(gdb) p info.status
$7 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id   Target Id                                          Frame
* 1    Thread 0x7ffff5ee55c0 (LWP 5066) "qemu-system-x86" fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
  2    Thread 0x7ffff5ee1700 (LWP 5070) "qemu-system-x86" (running)
  3    Thread 0x7ffff565f700 (LWP 5071) "qemu-system-x86" (running)
  5    Thread 0x7fffedfff700 (LWP 5075) "qemu-system-x86" migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1463
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffedfff700 (LWP 5075))]
#0  migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1463
1463    in ./migration/migration.c

(gdb) continue &
Continuing.
(gdb) info threads
  Id   Target Id                                          Frame
  1    Thread 0x7ffff5ee55c0 (LWP 5066) "qemu-system-x86" fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
  2    Thread 0x7ffff5ee1700 (LWP 5070) "qemu-system-x86" (running)
  3    Thread 0x7ffff565f700 (LWP 5071) "qemu-system-x86" (running)
* 5    Thread 0x7fffedfff700 (LWP 5075) "qemu-system-x86" (running)

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff5ee55c0 (LWP 5066))]
#0  fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
928     in ./migration/migration.c
(gdb) p (MigrationStatus) s.state
$8 = MIGRATION_STATUS_ACTIVE
(gdb) c

T3)

(qemu) info migrate
info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 0 milliseconds
(qemu)

Migration status is active, without any RAM statistics.

(qemu) quit
(gdb) quit

Terminal 1)

Ctrl-C




focal-proposed: PASS
--------------


$ sudo add-apt-repository -ys 'deb http://archive.ubuntu.com/ubuntu focal-proposed main'
$ sudo add-apt-repository -y 'deb http://ddebs.ubuntu.com/ubuntu focal-proposed main'

$ sudo apt install --yes qemu-system-x86 qemu-system-x86-dbgsym


$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:4.2-3ubuntu6.25

$ dpkg -s qemu-system-x86-dbgsym | grep Version:
Version: 1:4.2-3ubuntu6.25

...

$ apt source qemu

$ head -n1 qemu-*/debian/changelog
qemu (1:4.2-3ubuntu6.25) focal; urgency=medium

$ vim qemu-*/migration/migration.c

 915 static void fill_source_migration_info(MigrationInfo *info)
...
 926     case MIGRATION_STATUS_SETUP:
 927         info->has_status = true;
 928         info->has_total_time = false;
 929         break;

...




T1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

T2)

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  qemu-system-x86_64
  
(gdb) b migrate_set_state
Breakpoint 1 at 0x6d3b80: migrate_set_state. (2 locations)
(gdb) b migration/migration.c:928
Breakpoint 2 at 0x6d32ad: file ./migration/migration.c, line 928.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off

T3)

nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566949d8, old_state=0, new_state=1) at ./migration/migration.c:1464
1464    ./migration/migration.c: No such file or directory.
(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c
Continuing.
[New Thread 0x7fffedfff700 (LWP 6990)]
[New Thread 0x7fffed7fe700 (LWP 6991)]
[Thread 0x7fffedfff700 (LWP 6990) exited]

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1464
1464    in ./migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE
(gdb)

T3)

(qemu) info migrate

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
928     in ./migration/migration.c

(gdb) p (MigrationStatus) s.state
$6 = MIGRATION_STATUS_SETUP
(gdb) p info.status
$7 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id   Target Id                                          Frame
* 1    Thread 0x7ffff5ee55c0 (LWP 6983) "qemu-system-x86" fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
  2    Thread 0x7ffff5ee1700 (LWP 6987) "qemu-system-x86" (running)
  3    Thread 0x7ffff565f700 (LWP 6988) "qemu-system-x86" (running)
  5    Thread 0x7fffed7fe700 (LWP 6991) "qemu-system-x86" migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1464
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffed7fe700 (LWP 6991))]
#0  migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1464
1464    in ./migration/migration.c

(gdb) continue &
Continuing.

(gdb) info threads
  Id   Target Id                                          Frame
  1    Thread 0x7ffff5ee55c0 (LWP 6983) "qemu-system-x86" fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
  2    Thread 0x7ffff5ee1700 (LWP 6987) "qemu-system-x86" (running)
  3    Thread 0x7ffff565f700 (LWP 6988) "qemu-system-x86" (running)
* 5    Thread 0x7fffed7fe700 (LWP 6991) "qemu-system-x86" (running)
(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff5ee55c0 (LWP 6983))]
#0  fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
928     in ./migration/migration.c
(gdb) p (MigrationStatus) s.state
$8 = MIGRATION_STATUS_ACTIVE
(gdb) c

T3)

(qemu) info migrate
info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: setup
total time: 0 milliseconds

Status is now still 'SETUP' (which is not expected to have RAM statistics), not 'ACTIVE' (which is, and caused the issue).

(qemu) quit
(gdb) quit

Terminal 1)

Ctrl-C

Verification done for bionic-proposed.

bionic-updates: FAIL (status: active)

	(qemu) info migrate
	info migrate
	globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
	capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off
	Migration status: active
	total time: 0 milliseconds

bionic-proposed: PASS (status: setup)

	(qemu) info migrate
	info migrate
	globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
	capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off
	Migration status: setup
	total time: 0 milliseconds

Details:
=======

$ lsb_release -cs
bionic

bionic-updates: FAIL
--------------

$ curl http://ddebs.ubuntu.com/dbgsym-release-key.asc | sudo apt-key add -
OK
$ sudo add-apt-repository -y 'deb http://ddebs.ubuntu.com/ubuntu bionic-updates main'

$ sudo apt install --yes qemu-system-x86 qemu-system-x86-dbgsym

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:2.11+dfsg-1ubuntu7.41

$ dpkg -s qemu-system-x86-dbgsym | grep Version:
Version: 1:2.11+dfsg-1ubuntu7.41

...

$ sudo add-apt-repository -ys 'deb http://archive.ubuntu.com/ubuntu bionic-updates main'
$ apt source qemu
$ head -n1 qemu-*/debian/changelog
qemu (1:2.11+dfsg-1ubuntu7.41) bionic-security; urgency=medium

$ vim qemu-*/migration/migration.c
 594 MigrationInfo *qmp_query_migrate(Error **errp)
 ...
 603     case MIGRATION_STATUS_SETUP:
 604         info->has_status = true;
 605         info->has_total_time = false;
 606         break;
...

T1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

T2)

(gdb) b migrate_set_state
Breakpoint 1 at 0x58dd60: file ./migration/migration.c, line 1025.

(gdb) b migration/migration.c:603
Breakpoint 2 at 0x58d670: file ./migration/migration.c, line 603.

# Using 603 vs 606 as the code didn't break on 606 for some reason.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off

T3)

$ nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566ac6f0, old_state=0, new_state=1) at ./migration/migration.c:1025
1025    ./migration/migration.c: No such file or directory.
(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c
Continuing.
[New Thread 0x7fffcf9ff700 (LWP 4909)]
[New Thread 0x7fffcf1fe700 (LWP 4910)]
[Thread 0x7fffcf9ff700 (LWP 4909) exited]

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1025
1025    in ./migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

T3)

(qemu) info migrate

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:604
604     in ./migration/migration.c
(gdb) p (MigrationStatus) s.state
value has been optimized out
(gdb) p info.status
$1 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fc1cc0 (LWP 4916) "qemu-system-x86" qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:604
  2    Thread 0x7fffdf7ff700 (LWP 4920) "qemu-system-x86" (running)
  3    Thread 0x7fffdcffe700 (LWP 4921) "qemu-system-x86" (running)
  5    Thread 0x7fffcf9ff700 (LWP 4924) "qemu-system-x86" migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1025

(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffcf9ff700 (LWP 4924))]
#0  migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1025
1025    in ./migration/migration.c
(gdb) continue &
Continuing.

(gdb) info threads
  Id   Target Id         Frame
  1    Thread 0x7ffff7fc1cc0 (LWP 4916) "qemu-system-x86" qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:604
  2    Thread 0x7fffdf7ff700 (LWP 4920) "qemu-system-x86" (running)
  3    Thread 0x7fffdcffe700 (LWP 4921) "qemu-system-x86" (running)
* 5    Thread 0x7fffcf9ff700 (LWP 4924) "qemu-system-x86" (running)

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7fc1cc0 (LWP 4916))]
#0  qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:604
604     in ./migration/migration.c
(gdb) p (MigrationStatus) s.state
value has been optimized out
(gdb) c

T3)

(qemu) info migrate
info migrate
globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off
Migration status: active
total time: 0 milliseconds

Migration status is active, without any RAM statistics.



bionic-proposed: PASS
---------------

$ sudo add-apt-repository -ys 'deb http://archive.ubuntu.com/ubuntu bionic-proposed main'
$ sudo add-apt-repository -y 'deb http://ddebs.ubuntu.com/ubuntu bionic-proposed main'

$ sudo apt install --yes qemu-system-x86 qemu-system-x86-dbgsym

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:2.11+dfsg-1ubuntu7.42

$ dpkg -s qemu-system-x86-dbgsym | grep Version:
Version: 1:2.11+dfsg-1ubuntu7.42

$ rm -rf qemu*
$ apt source qemu

$ head -n1 qemu-*/debian/changelog
qemu (1:2.11+dfsg-1ubuntu7.42) bionic; urgency=medium

$ vim qemu-*/migration/migration.c
 594 MigrationInfo *qmp_query_migrate(Error **errp)
...
 604     case MIGRATION_STATUS_SETUP:
 605         info->has_status = true;
 606         info->has_total_time = false;
 607         break;


T1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

T2)

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  qemu-system-x86_64

(gdb) b migrate_set_state
Breakpoint 1 at 0x58de30: file ./migration/migration.c, line 1026.

(gdb) b migration/migration.c:604
Breakpoint 2 at 0x58d748: file ./migration/migration.c, line 604.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off


T3)

nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566ac6f0, old_state=0, new_state=1) at ./migration/migration.c:1026
1026    ./migration/migration.c: No such file or directory.
(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c
Continuing.
[New Thread 0x7fffcf9ff700 (LWP 6525)]
[New Thread 0x7fffcf1fe700 (LWP 6526)]
[Thread 0x7fffcf9ff700 (LWP 6525) exited]

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1026
1026    in ./migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

T3)

(qemu) info migrate

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:605
605     in ./migration/migration.c
(gdb) p (MigrationStatus) s.state
value has been optimized out
(gdb) p info.status
$5 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fc1cc0 (LWP 6518) "qemu-system-x86" qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:605
  2    Thread 0x7fffdf7ff700 (LWP 6522) "qemu-system-x86" (running)
  3    Thread 0x7fffdcffe700 (LWP 6523) "qemu-system-x86" (running)
  5    Thread 0x7fffcf1fe700 (LWP 6526) "qemu-system-x86" migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1026
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffcf1fe700 (LWP 6526))]
#0  migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1026
1026    in ./migration/migration.c
(gdb) continue &
Continuing.
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 0x7ffff7fc1cc0 (LWP 6518) "qemu-system-x86" qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:605
  2    Thread 0x7fffdf7ff700 (LWP 6522) "qemu-system-x86" (running)
  3    Thread 0x7fffdcffe700 (LWP 6523) "qemu-system-x86" (running)
* 5    Thread 0x7fffcf1fe700 (LWP 6526) "qemu-system-x86" (running)

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7fc1cc0 (LWP 6518))]
#0  qmp_query_migrate (errp=errp@entry=0x0) at ./migration/migration.c:605
605     in ./migration/migration.c
(gdb) p (MigrationStatus) s.state
value has been optimized out

(gdb) c

T3)

(qemu) info migrate
info migrate
globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off
Migration status: setup
total time: 0 milliseconds

Status is now still 'SETUP' (which is not expected to have RAM statistics), not 'ACTIVE' (which is, and caused the issue).

(qemu) quit
(gdb) quit

Terminal 1)

Ctrl-C


I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions at the moment.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on the previous update.


This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.7

---------------
qemu (1:6.2+dfsg-2ubuntu6.7) jammy; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)
  * d/rules: move "Disable LTO on non-amd64" before buildflags.mk on Jammy.

  [ Michal Maloszewski ]
  * d/rules: Disable LTO on non-amd 64 architectures to prevent QEMU
    coroutines from failing (LP: #1921664)

 -- Mauricio Faria de Oliveira <email address hidden>  Mon, 06 Mar 2023 17:00:46 -0300

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.25

---------------
qemu (1:4.2-3ubuntu6.25) focal; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)

 -- Mauricio Faria de Oliveira <email address hidden>  Thu, 02 Mar 2023 18:07:21 -0300

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.42

---------------
qemu (1:2.11+dfsg-1ubuntu7.42) bionic; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)

 -- Mauricio Faria de Oliveira <email address hidden>  Thu, 02 Mar 2023 18:26:12 -0300

Marking the UCA tasks as Invalid, as apparently not needed; please revert otherwise.

Hello Brett, or anyone else affected,

Accepted qemu into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verification done on ussuri-proposed.
Steps explained in previous comments.

The migration status with the synthetic reproducer in GDB
is now still 'SETUP' (which is not expected to have RAM statistics),
instead of 'ACTIVE' (which is, and caused the issue):

(qemu) info migrate
...
Migration status: setup

...

$ lsb_release -cs
bionic

$ apt policy qemu-system-x86
qemu-system-x86:
  Installed: 1:4.2-3ubuntu6.27~cloud0
  Candidate: 1:4.2-3ubuntu6.27~cloud0
  Version table:
 *** 1:4.2-3ubuntu6.27~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
...

$ file $(which qemu-system-x86_64) | grep -o 'BuildID.*,'
BuildID[sha1]=82a4159294ae653e770be24bbcfbb35703e60005,

(Corey provided the .ddeb packages which is not yet exposed/published in PPAs/archive.)

$ dpkg-deb -c qemu-system-x86-dbgsym_4.2-3ubuntu6.27~cloud0_amd64.ddeb | fgrep .debug
-rw-r--r-- root/root  21271712 2023-10-26 14:08 ./usr/lib/debug/.build-id/48/bd78ceee4a669d37efd9ac8d851947205de4f7.debug
-rw-r--r-- root/root  21321832 2023-10-26 14:08 ./usr/lib/debug/.build-id/82/a4159294ae653e770be24bbcfbb35703e60005.debug

$ sudo apt install ./qemu-system-x86-dbgsym_4.2-3ubuntu6.27~cloud0_amd64.ddeb

$ apt source qemu

$ head -n1 qemu-4.2/debian/changelog
qemu (1:4.2-3ubuntu6.27~cloud0) bionic-ussuri; urgency=medium

 915 static void fill_source_migration_info(MigrationInfo *info)
...
 926     case MIGRATION_STATUS_SETUP:
 927         info->has_status = true;
 928         info->has_total_time = false;
 929         break;

Terminal 1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

$ gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  qemu-system-x86_64
...
Reading symbols from qemu-system-x86_64...Reading symbols from /usr/lib/debug/.build-id/82/a4159294ae653e770be24bbcfbb35703e60005.debug...done.
done.

(gdb) b migrate_set_state
Breakpoint 1 at 0x6ba8c0: file ./migration/migration.c, line 1464.

(gdb) b migration/migration.c:928
Breakpoint 2 at 0x6b9fb3: file ./migration/migration.c, line 928.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off
...

Terminal 3)

$ nc 127.0.0.1 3333
QEMU 4.2.1 monitor - type 'help' for more information
(qemu) migrate -d tcp:127.0.0.1:4444

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566a11d8, old_state=0, new_state=1) at ./migration/migration.c:1464
1464    ./migration/migration.c: No such file or directory.

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE

(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP

(gdb) c
Continuing.
...

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566a11d8, old_state=1, new_state=4) at ./migration/migration.c:1464
1464    in ./migration/migration.c

(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP

(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

(gdb)

Terminal 3)

(qemu) info migrate

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x5555572d29b0) at ./migration/migration.c:928
928     in ./migration/migration.c

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fcdcc0 (LWP 1477) "qemu-system-x86" fill_source_migration_info (info=0x5555572d29b0) at ./migration/migration.c:928
  2    Thread 0x7fffe61ff700 (LWP 1481) "qemu-system-x86" (running)
  3    Thread 0x7fffe59fe700 (LWP 1482) "qemu-system-x86" (running)
  5    Thread 0x7fffdd7fe700 (LWP 1485) "qemu-system-x86" migrate_set_state (state=0x5555566a11d8, old_state=1, new_state=4) at ./migration/migration.c:1464
  
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffdd7fe700 (LWP 1485))]
#0  migrate_set_state (state=0x5555566a11d8, old_state=1, new_state=4) at ./migration/migration.c:1464
1464    in ./migration/migration.c

(gdb) continue &
Continuing.

(gdb) info threads
  Id   Target Id         Frame
  1    Thread 0x7ffff7fcdcc0 (LWP 1477) "qemu-system-x86" fill_source_migration_info (info=0x5555572d29b0) at ./migration/migration.c:928
  2    Thread 0x7fffe61ff700 (LWP 1481) "qemu-system-x86" (running)
  3    Thread 0x7fffe59fe700 (LWP 1482) "qemu-system-x86" (running)
* 5    Thread 0x7fffdd7fe700 (LWP 1485) "qemu-system-x86" (running)

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7fcdcc0 (LWP 1477))]
#0  fill_source_migration_info (info=0x5555572d29b0) at ./migration/migration.c:928
928     in ./migration/migration.c

(gdb) c
Continuing.

Terminal 3)

(qemu) info migrate
info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: setup
total time: 0 milliseconds

Status is now still 'SETUP' (which is not expected to have RAM statistics), not 'ACTIVE' (which is, and caused the issue).

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.


This bug was fixed in the package qemu - 1:4.2-3ubuntu6.27~cloud0
---------------

 qemu (1:4.2-3ubuntu6.27~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 qemu (1:4.2-3ubuntu6.27) focal-security; urgency=medium
 .
   * SECURITY UPDATE: user-after-free issue
     - debian/patches/CVE-2022-1050.patch: Protect against buggy or
       malicious guest driver
     - CVE-2022-1050
   * SECURITY UPDATE: Out-of-bounds read
     - debian/patches/CVE-2022-4144-*.patch: Have qxl_log_command Return
       early if no log_cmd handler; Document qxl_phys2virt(); Pass requested
       buffer size to qxl_phys2virt(); Avoid buffer overrun in qxl_phys2virt;
       Assert memory slot fits in preallocated MemoryRegion
     - CVE-2022-4144
   * SECURITY UPDATE: reentrancy problem
     - debian/patches/CVE-2023-0330.patch: Fix reentrancy issues in the LSI
       controller
     - CVE-2023-0330
 .
 qemu (1:4.2-3ubuntu6.26) focal; urgency=medium
 .
   * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
     avoid timer issues in s390x secure execution guests (LP: #1999885)
 .
 qemu (1:4.2-3ubuntu6.25) focal; urgency=medium
 .
   [ Brett Milford ]
   * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
     error 'migration was active, but no RAM info was set' (LP: #1994002)
 .
   [ Mauricio Faria de Oliveira ]
   * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
     to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)