other: 0.626 device: 0.586 semantic: 0.555 graphic: 0.532 network: 0.518 socket: 0.501 boot: 0.486 KVM: 0.470 vnc: 0.465 [Qemu-devel] [Bug] Docs build fails at interop.rst https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31 (Rawhide) uname - a Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431 allows for the build to occur Regards Aarushi Mehta On 5/20/19 7:30 AM, Aarushi Mehta wrote: > https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw > running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31 > (Rawhide) > > uname - a > Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32 > UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > > Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431 > allows for the build to occur > > Regards > Aarushi Mehta > > Ah, dang. The blocks aren't strictly conforming json, but the version I tested this under didn't seem to care. Your version is much newer. (I was using 1.7 as provided by Fedora 29.) For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead, which should at least turn off the "warnings as errors" option, but I don't think that reverting -n will turn off this warning. I'll try to get ahold of this newer version and see if I can't fix it more appropriately. --js On 5/20/19 12:37 PM, John Snow wrote: > > > On 5/20/19 7:30 AM, Aarushi Mehta wrote: > > https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw > > running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31 > > (Rawhide) > > > > uname - a > > Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32 > > UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > > > > Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431 > > allows for the build to occur > > > > Regards > > Aarushi Mehta > > > > > > Ah, dang. The blocks aren't strictly conforming json, but the version I > tested this under didn't seem to care. Your version is much newer. (I > was using 1.7 as provided by Fedora 29.) > > For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead, > which should at least turn off the "warnings as errors" option, but I > don't think that reverting -n will turn off this warning. > > I'll try to get ahold of this newer version and see if I can't fix it > more appropriately. > > --js > ...Sigh, okay. So, I am still not actually sure what changed from pygments 2.2 and sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx by default always tries to do add a filter to the pygments lexer that raises an error on highlighting failure, instead of the default behavior which is to just highlight those errors in the output. There is no option to Sphinx that I am aware of to retain this lexing behavior. (Effectively, it's strict or nothing.) This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the build works with our malformed json. There are a few options: 1. Update conf.py to ignore these warnings (and all future lexing errors), and settle for the fact that there will be no QMP highlighting wherever we use the directionality indicators ('->', '<-'). 2. Update bitmaps.rst to remove the directionality indicators. 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON. 4. Update bitmaps.rst to remove the "json" specification from the code block. This will cause sphinx to "guess" the formatting, and the pygments guesser will decide it's Python3. This will parse well enough, but will mis-highlight 'true' and 'false' which are not python keywords. This approach may break in the future if the Python3 lexer is upgraded to be stricter (because '->' and '<-' are still invalid), and leaves us at the mercy of both the guesser and the lexer. I'm not actually sure what I dislike the least; I think I dislike #1 the most. #4 gets us most of what we want but is perhaps porcelain. I suspect if we attempt to move more of our documentation to ReST and Sphinx that we will need to answer for ourselves how we intend to document QMP code flow examples. --js On Mon, May 20, 2019 at 05:25:28PM -0400, John Snow wrote: > > > On 5/20/19 12:37 PM, John Snow wrote: > > > > > > On 5/20/19 7:30 AM, Aarushi Mehta wrote: > >> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw > >> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31 > >> (Rawhide) > >> > >> uname - a > >> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32 > >> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > >> > >> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431 > >> allows for the build to occur > >> > >> Regards > >> Aarushi Mehta > >> > >> > > > > Ah, dang. The blocks aren't strictly conforming json, but the version I > > tested this under didn't seem to care. Your version is much newer. (I > > was using 1.7 as provided by Fedora 29.) > > > > For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead, > > which should at least turn off the "warnings as errors" option, but I > > don't think that reverting -n will turn off this warning. > > > > I'll try to get ahold of this newer version and see if I can't fix it > > more appropriately. > > > > --js > > > > ...Sigh, okay. > > So, I am still not actually sure what changed from pygments 2.2 and > sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx > by default always tries to do add a filter to the pygments lexer that > raises an error on highlighting failure, instead of the default behavior > which is to just highlight those errors in the output. There is no > option to Sphinx that I am aware of to retain this lexing behavior. > (Effectively, it's strict or nothing.) > > This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the > build works with our malformed json. > > There are a few options: > > 1. Update conf.py to ignore these warnings (and all future lexing > errors), and settle for the fact that there will be no QMP highlighting > wherever we use the directionality indicators ('->', '<-'). > > 2. Update bitmaps.rst to remove the directionality indicators. > > 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON. > > 4. Update bitmaps.rst to remove the "json" specification from the code > block. This will cause sphinx to "guess" the formatting, and the > pygments guesser will decide it's Python3. > > This will parse well enough, but will mis-highlight 'true' and 'false' > which are not python keywords. This approach may break in the future if > the Python3 lexer is upgraded to be stricter (because '->' and '<-' are > still invalid), and leaves us at the mercy of both the guesser and the > lexer. > > I'm not actually sure what I dislike the least; I think I dislike #1 the > most. #4 gets us most of what we want but is perhaps porcelain. > > I suspect if we attempt to move more of our documentation to ReST and > Sphinx that we will need to answer for ourselves how we intend to > document QMP code flow examples. Writing a custom lexer that handles "<-" and "->" was simple (see below). Now, is it possible to convince Sphinx to register and use a custom lexer? $ cat > /tmp/lexer.py < *', Generic.Prompt), (r' *<- *', Generic.Output), ] } class QMPExampleLexer(DelegatingLexer): def __init__(self, **options): super(QMPExampleLexer, self).__init__(JsonLexer, QMPExampleMarkersLexer, Error, **options) EOF $ pygmentize -l /tmp/lexer.py:QMPExampleLexer -x -f html < { "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", "target": "drive0.inc0.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } <- { "return": {} } EOF 
    -> 
{
         "execute": 
"drive-backup",
         "arguments": 
{
           "device": 
"drive0",
           "bitmap": 
"bitmap0",
           "target": 
"drive0.inc0.qcow2",
           "format": 
"qcow2",
           "sync": 
"incremental",
           "mode": 
"existing"
         }
       }

    <- { "return": {} }
$ -- Eduardo On 5/20/19 7:04 PM, Eduardo Habkost wrote: > On Mon, May 20, 2019 at 05:25:28PM -0400, John Snow wrote: > > > > > > On 5/20/19 12:37 PM, John Snow wrote: > >> > >> > >> On 5/20/19 7:30 AM, Aarushi Mehta wrote: > >>> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw > >>> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31 > >>> (Rawhide) > >>> > >>> uname - a > >>> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32 > >>> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > >>> > >>> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431 > >>> allows for the build to occur > >>> > >>> Regards > >>> Aarushi Mehta > >>> > >>> > >> > >> Ah, dang. The blocks aren't strictly conforming json, but the version I > >> tested this under didn't seem to care. Your version is much newer. (I > >> was using 1.7 as provided by Fedora 29.) > >> > >> For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead, > >> which should at least turn off the "warnings as errors" option, but I > >> don't think that reverting -n will turn off this warning. > >> > >> I'll try to get ahold of this newer version and see if I can't fix it > >> more appropriately. > >> > >> --js > >> > > > > ...Sigh, okay. > > > > So, I am still not actually sure what changed from pygments 2.2 and > > sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx > > by default always tries to do add a filter to the pygments lexer that > > raises an error on highlighting failure, instead of the default behavior > > which is to just highlight those errors in the output. There is no > > option to Sphinx that I am aware of to retain this lexing behavior. > > (Effectively, it's strict or nothing.) > > > > This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the > > build works with our malformed json. > > > > There are a few options: > > > > 1. Update conf.py to ignore these warnings (and all future lexing > > errors), and settle for the fact that there will be no QMP highlighting > > wherever we use the directionality indicators ('->', '<-'). > > > > 2. Update bitmaps.rst to remove the directionality indicators. > > > > 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON. > > > > 4. Update bitmaps.rst to remove the "json" specification from the code > > block. This will cause sphinx to "guess" the formatting, and the > > pygments guesser will decide it's Python3. > > > > This will parse well enough, but will mis-highlight 'true' and 'false' > > which are not python keywords. This approach may break in the future if > > the Python3 lexer is upgraded to be stricter (because '->' and '<-' are > > still invalid), and leaves us at the mercy of both the guesser and the > > lexer. > > > > I'm not actually sure what I dislike the least; I think I dislike #1 the > > most. #4 gets us most of what we want but is perhaps porcelain. > > > > I suspect if we attempt to move more of our documentation to ReST and > > Sphinx that we will need to answer for ourselves how we intend to > > document QMP code flow examples. > > Writing a custom lexer that handles "<-" and "->" was simple (see below). > > Now, is it possible to convince Sphinx to register and use a custom lexer? > Spoilers, yes, and I've sent a patch to list. Thanks for your help!