Date: 14 March 91 Message No: 037 To: TeX implementors and distributors From: Barbara Beeton Subject: Messages from DEK, part 2 ************************************************************************ >>> inconsistent expansion of \string, etc.; \newlinechar [ dek: Here he has come across exactly what I think I fixed in the _other_ change made to 3.14a. I am contacting him directly about this. ] Date: Mon, 4 Nov 91 10:42:40 +0100 From: Bernd Raichle Subject: TeX.web 3.14: Bug Report #1 (+ Fix Proposal): \string, \special, ... Dear Ms. Beeton, I have found two bugs in TeX.Web 3.14: Bug #1: Incorrect "expansion" of unprintable chars for |selector=new_string| Bug #2: Incorrect handling of \newlinechar (I'll send a second mail with the report of bug #2.) The first bug looks like an inconsistent behaviour of \string (and \meaning, \fontname, \jobname, \special) in all TeX versions. Example: 1. \catcode`\^^Y=12 \string^^Y 2. \catcode`\^^Y=13 \string^^Y TeX 3.14 prints line 1 always as character "19 of the current font. Line 2 is printed either as char"19 or as three characters ^^Y, dependent on the TeX implementation! Other examples: 3. If ^^Y has one of the catcodes 1-4, 6-8, 11 or 12 |\string ^^Y| is printed like |\char"19|. 4. With \escapechar="19, the output of |\string\relax| is either like |^^Yrelax| or |\char"19 relax|. 5. The same is true for \catcode`\^^Y=11 \string\test^^Ytest 6. \newlinechar=`\^^Y \message{^^Y \string ^^Y} outputs two newlines only, if ^^Y is a printable character. (Remark: I have discovered the second bug with a variation of this example.) 7. The same is true for things like \special{^^Y}. 8. etc. for other characters (e.g. national characters)! An TeX implementor can (and should) change section 21 and 22 (character set translations) and section 49 (unprintable characters) to make TeX more user friendly. The predicate in section 49 specifies if a characters is printable or unprintable. Example: \ss (= char "19 in CM fonts) should be output as a single character instead of ^^Y. An implementor will change section 49 to exclude this character (i.e., to make this character printable). [With TeX 3 this is done for more and more characters on more and more machines with extended character set (especially if the new DC/EC fonts with national characters are available).] This implementation dependency is no problem (*), if the output is printed on a terminal or written in a file (log file or output stream). [(*) see the second bug report for some problems] But it can be a problem, if it is "printed" for further `internal' use (i.e., with |selector:=new_string|). Example: \def\a{\expandafter\b\string} \def\b#1{{\tt #1}} The output (in the dvi file) is implementation dependent, if \a is called with national characters, control sequence names with national characters, etc. The dvi file will also be different, if someone uses \special commands with national characters. E.g. \special{^^Y} should put one byte with value "19 into the dvi file, but for most TeX versions the three characters "^^Y" are written. Fix proposal: |print| and |slow_print| (\S 59,60) should print a single character without using |str_pool| if |selector = new_string|. (I.e., the `expansion' of unprintable characters to ^^X or ^^xx is inhibited.) procedure print(s : integer); label exit; var j : poolpointer; begin if s >= str_ptr then s:="???" else if s < 256 then if s < 0 then s:="???" + else if selector = new_string then + begin print_char(s); return + end; else if (s = new_line_char) then if selector < pseudo then begin print_ln; return end; j:=str_start[s]; while j < str_start[s+1] do begin print_char(so(str_pool[j])); incr(j); end; exit: end; ... and the same for |slow_print|. Additional changes: |issue_message| (used by \message and \errmessage) calls |print| and |print_err|. Because the string argument can contain unprintable character with the fix above, it has to be replaced by |slow_print|. Additionally the |print_err| call has to be replaced by a call to a new procedure |slow_print_err|, which calls |slow_print| instead of |print|. I don't know, if it's better to print strings like |job_name|, |log_name|, |font_name()|... with |slow_print| (This could be a third bug?). If this is true, the string |formatident| has to be printed with |slow_print|, because it incorporates |job_name|. All other strings `printed' with |selector:=new_string| are either made out of internal strings and integers or they are used only temporary (|special_out|, |ship_out|) or printed with |print_esc| (|font_id_text()|), so they do not need further changes. Possible problems: - Perhaps some macro writers assume the current behaviour of \string et.al. I.e., they assume that \string expands to a fixed set of "printable" characters only (e.g. for verbatim text like {\tt \string\foo}), but this set of printable characters changes if section 49 is changed. - The behaviour of \message and \errmessage will change: \newlinechar has an effect to these primitives (the TeXbook describes only the effect for \write). - I have made and tested the changes above... and this changed TeX passes the Trip-Test without problems. Perhaps it would be a good idea to add checks for \special and \string with arguments containing unprintable characters to `trip.tex'. Additionally there should be a short note, that the Trip-Test can produce a different output, if section 49 is changed. The output is different if one of the characters ^^@, ..., ^^82, ... is printable. Yours, Bernd Raichle __________________________________________________________________________ Bernd Raichle, Student der Universit"at Stuttgart | "Le langage est source home: Stettener Str. 73, D-W-7300 Esslingen, FRG | de malentendus" email: raichle@azu.informatik.uni-stuttgart.de | (A. de Saint-Exupery) ------- Date: Mon, 4 Nov 91 10:43:42 +0100 From: Bernd Raichle Subject: TeX.web 3.14: Bug Report #2 (+ Fix Proposal): \newlinechar Dear Ms. Beeton, here is the report of the second bug. I have found this bug when experimenting with examples (to test the behaviour of \message and \write) and possible fixes for the first bug. The second bug deals with \newlinechar and the inconsistent behaviour of \message and \write when it is set to some "exotic" values. Example: \catcode`\^^Y=11 \edef\test{\string\t^^Yt} % (1) \newlinechar=`\^ \message{^^Y ^ \string\^^Y ^ \test} \immediate\write0{^^Y ^ \string\^^Y ^ \test} % (2) \newlinechar=`\^^Y \escapechar=`\^^Y \message{^^Y \string ^^Y \relax} \immediate\write0{^^Y \string ^^Y \relax} % (3) \newlinechar=`\o The output of this example is implementation dependent! The output differs, if section 49 of TeX.web is changed to make the character ^^Y printable. 1. In part (1) of the example, the character ^^Y is printed as two newlines followed by a `Y' (if ^^Y is unprintable). The same aplies for all other ^^Y characters, even for ^^Y characters inside control sequences. 2. The same ^^Y character is printed as character |xchr["19]|, if section 49 is changed to make it printable. 3. The output of \write and \message in part (2) differs, if ^^Y is unprintable. \message produces three |^^Y|, \write prints three newlines. 4. There is no difference between \write and \message in part (1) and (2), if character ^^Y is made printable. (Remark: the TeXbook doesn't mention \newlinechar w.r.t. \message, so one could assume that changing \newlinechar will not change the output of \message (and \errmessage).) What behaviour should be correct for |print|, |slow_print|, ... w.r.t. \newlinechar, if something is printed to a file or the terminal? Here are my wishes: 1) if one of these functions is called with the current \newlinechar, it should output a newline. 2) if a tokenlist printed with \write, \message, \errmessage contains a \newlinechar, it should be printed as a newline. 3) _all_ other characters should never produce a newline character (even if there are unprintable characters and \newlinechar = `\^ !) 4) control sequence names containing a character equal to the current \newlinechar should not produce a newline (a user should use \string, if the newline character in the csname should produce a newline) 5) all fixed strings printed to the log file should never produce a newline. I think error messages in the log file should be readable for all values of \newlinechar! I have added part (3) in my example to show the current (mis-)behaviour. [ dek: Here we have a disagreement ... I don't believe it is necessary to support every possible \newlinechar in an optimum way; here the simpler rule without exceptions wins ] Fix proposal: The changes below needs the changes in |issue_message|, I have described in my first bug report (replace call of |print| in this function with |slow_print|, introduce new function |slow_print_err| and replace |print_err| in |issue_message| with a call of this funtion). Otherwise \message and \errmessage produce incorrect output for unprintable characters. I have made the following changes to |print_char|, |print| and |slow_print|: (* the old |print_char| procedure: *) procedure print_the_char ( s : ASCII_code ) (* special handling of current new_line_char deleted *) case selector of ... etc, etc. old code of |print_char|... end; procedure print_char ( s : ASCII_code ) label exit; begin if then if selector < pseudo then begin print_ln; return; end; print_the_char(s); exit: end; procedure print ( s : integer ); label exit; var j : poolpointer; begin if s >= str_ptr then s:="???" else if s < 256 then if s < 0 then s:="???" (* bug fix from bug report #1: *) + else if selector = new_string then + begin print_the_char(s); return + end; else if (s = new_line_char) then if selector < pseudo then begin print_ln; return end; j:=str_start[s]; while j < str_start[s+1] do (* to fulfill wish #4: *) ! begin print_the_char(so(str_pool[j])); incr(j); end; exit: end; procedure slow_print ( s : integer ); label exit; var j : poolpointer; begin if s >= str_ptr then s:="???" else if s < 256 then (* for wish #3: *) ! begin print(s); return end; j:=str_start[s]; while j < str_start[s+1] do begin print(so(str_pool[j])); incr(j); end; exit: end; I was very astonished that (after I had applied the changes) the resulted "TeX" passes the Trip-Test! Possible Problems(?): - \newlinechar handling is done in \write, \message or \errmessage. \write uses |token_show| to output the tokenlist and this is done by calls of |print_char| or |print| with single characters or by calls of |print_cs|. \message and \errmessage uses |slow_print| or |slow_err_print|, which handles newline characters at the `toplevel' of the string. - There should be additional changes to satisfy my wish #5 (I don't know, if it is enough to change the |print_char| calls in the procedure |show_token_list| to call a |print_char| procedure with \newlinechar handling. And the calls of |print_char| in all other places are calls to a |print_char| version without \newlinechar handling (e.g. the |print_the_char| above).) Yours, Bernd Raichle <<< end inconsistent expansion of \string, etc.; \newlinechar ************************************************************************ >>> \halign + box assignment within displays Date: Thu, 21 Nov 91 16:55:55 GMT From: Chris Thompson Cc: Robert Hunt Subject: TeX bug in math alignments (Robert Hunt) Barbara, I received the following bug report from Robert Hunt > Accepted: 12:02:31 21 Nov 91 > Submitted: 23:36:12 19 Nov 91 > IPMessageId: A4D2794E8E47F040 > From: Robert Hunt > To: cet1 > Subject: Bug (sigh) in TeX > > Here's a bug in TeX: or possibly in the TeXbook, though I'd much > prefer it to be considered the former! The input > > $$\halign{#\cr x\cr}\global\setbox1=\hbox{y}$$ > > generates a "Missing $$ inserted" error just before the "y" token. > (Pressing RETURN then generates an "I can't go on meeting you like > this" error, and TeX bombs out.) This appears to be because TeX > objects to having horizontal mode material to process after an > alignment in displayed maths mode. However, page 291 of the TeXbook > clearly states that optional may follow the closing > brace of an \halign in display maths mode, and \setbox certainly > counts as an ! It is definitely a bug in TeX: here's a log file (duplicated on Unix TeX, by the way) --------------------start log file This is TeX, Phoenix Version 3.14:0.75 (preloaded format=plain 90.7.18) 21 NOV 1991 14:17 **sysin (&TEXR ! Missing $$ inserted. y l.1 $$\halign{#\cr x\cr}\global\setbox1=\hbox{y }$$ ? h Displays can use special alignments (like \eqalignno) only if nothing but the alignment itself is between $$'s. ? ! I can't go on meeting you like this. y l.1 $$\halign{#\cr x\cr}\global\setbox1=\hbox{y }$$ One of your faux pas seems to have wounded me deeply... [ dek: ^^^^^^^^^^^^^^^^^ how true ] in fact, I'm barely conscious. Please fix it and try again. No pages of output. --------------------end log file The problem is that the state of the world when |do_assignments| in module 1206 returns can be quite different from when it is called, in respect of the mode nest and the save stack in particular. It occurred to me that one should be able to create a simlar effect with the call in |make_accent| (module 1123), and indeed, one can: --------------------start log file This is TeX, Phoenix Version 3.14:0.75 (preloaded format=plain 90.7.18) 21 NOV 1991 15:05 **sysin (&TEXS ! This can't happen (vpack). } l.1 \accent"7F\global\setbox1=\vbox{\vskip1in} z I'm broken. Please show this to someone who can fix can fix No pages of output. --------------------end log file [ dek: $327.68 made out to Hunt ] Can you add this to Don's input queue, please? [ dek: As Chris undoubtedly knows, this is not at all easy to fix. I think it would be far too risky trying to allow arbitrary boxes to be assigned after \halign in display or between \accent and the accentee ... making _fin_align_ and _accent_ recursive would open up a huge can of worms and would make \afterassignment even murkier than it is now. Therefore I am modifying the TeXbook to disallow such things [which nobody has ever used because they never have worked] and modifying TeX to detect this prohibited activity ] Chris Thompson Cambridge University Computing Service JANET: cet1@uk.ac.cam.phx Internet: cet1@phx.cam.ac.uk <<< end \halign + box assignment within displays ************************************************************************ >>> positive/negative xn_over_d, tex.web section 107 Date: Thu, 21 Nov 91 18:38:51 GMT From: CET1@phoenix.cambridge.ac.uk Cc: Wayne Sullivan Subject: Wayne Sullivan's new TeX bug Barbara, Here is the promised bug report from Wayne Sullivan. I don't think I have anything useful to add to the messages below. I can produce backtraces very like Wayne's. The calls can come from |math_kern| as well as from |math_glue|. > Accepted: 16:47:07 15 Nov 91 > Submitted: 14:42:13 15 Nov 91 > From: "Wayne G. Sullivan" > To: Chris Thompson > Subject: possible bug in TeX In TeX WEB section 107, Knuth claims that for use of the procedure xn_over_d, n and d are always nonnegative. Whether the actual code makes use of this in section 107, I am not sure. However, some implementations utilize the assumed positivity in optimizing this section. In sections 716 and 717 xn_over_d is used in context with cur_mu. Though one would expect cur_mu to be nonnegative, I see no place where this is assured: I think one could simply change the corresponding font parameter to a negative value. Wayne Sullivan [ dek: $327.68 \S716-7 ] > Submitted: 17:18:39 15 Nov 91 > IPMessageId: A4CD1EF77B402240 > From: Chris Thompson > To: "Wayne G. Sullivan" > Subject: Re: [possible bug in TeX] I think you are right. |cur_mu| is set in section 703 and there is no reason why it has to be positive. Hence |f| in both sections 716 and 717 can be negative (as |x_over_n| gives a negative remainder when x and n have opposite signs). |xn_over_d| is *not* robust against negative n, quite apart from the promises in sections 107 and 99, because it causes u to be negative, and so (a) you can't assume u=d*(u div d)+(u mod d) in ANSI Pascal any longer, & (b) the test for (u div d) being too large would need to be supplemented by one for it being too small. I would like to construct a hard example of obvious bad results before passing this to Barbara & Don, though. Chris Thompson > Accepted: 14:57:26 18 Nov 91 > Submitted: 09:55:30 18 Nov 91 > From: "Wayne G. Sullivan" > To: Chris Thompson > Subject: xn_over_d I grabbed the pascal code for xn_over_d out of my change file to compare the result with an optimized version. On tangling I discovered that the intermediate values t,u,v are declared as nonnegative_integer. Hence a range checking compiler will choke on a negative cur_mu: Can DEK escape from that? > Accepted: 15:38:21 18 Nov 91 > Submitted: 09:32:41 18 Nov 91 > From: "Wayne G. Sullivan" > To: Chris Thompson > Subject: xn_over_d The use of xn_over_d with the possibly negative cur_mu only affects the the lower precision term in a `split' precision calculation so that the problem of overflow arises only if the input values are beyond the allowed values, i.e., with valid input values no overflow arises. Thus the algorithm will give the correct answer even though cur_mu is negative. However, this does depend on the div procedure satisfying negation symmetry: (-n) div d =-(n div d) when n,d>0. At any rate the documentation needs to be corrected so that optimized versions of the code will also give the correct result. > Accepted: 17:40:28 18 Nov 91 > Submitted: 11:05:39 18 Nov 91 > From: "Wayne G. Sullivan" > To: Chris Thompson > Subject: xnod Here is an example of the xn_over_d problem. Sorry about the antique version of TeX currently on our VM/CMS: I use it only to illustrate the problem. The TeX file: \showboxdepth100 \showboxbreadth100 \scrollmode \showthe\fontdimen6 \tensy \fontdimen6 \tensy=-10pt \showthe\fontdimen6 \tensy $$a\;a$$ \showlists \end The LOG file (errors were reported on the terminal). Look for negative glue. This is TeX, VM/CMS Version 2.1 (preloaded format=PLAIN 87.5.19) 18 NOV 1991 11 **xnod (xnod.tex.* > 10.00002pt. l.4 \showthe\fontdimen6 \tensy > -10.0pt. l.6 \showthe\fontdimen6 \tensy ### horizontal mode entered at line 7 spacefactor 1000 ### vertical mode entered at line 0 ### current page: \glue(\topskip) 10.0 \hbox(0.0+0.0)x469.75499, glue set 449.75488fil .\hbox(0.0+0.0)x20.0 .\penalty 10000 .\glue(\parfillskip) 0.0 plus 1.0fil .\glue(\rightskip) 0.0 \penalty 10000 \glue(\abovedisplayshortskip) 0.0 plus 3.0 \glue(\baselineskip) 7.69446 \hbox(4.30554+0.0)x7.79407, shifted 230.98047 .\teni a .\glue -2.77771 plus -2.77771 .\teni a \penalty 0 \glue(\belowdisplayshortskip) 7.0 plus 3.0 minus 4.0 total height 29.0 plus 6.0 minus 4.0 goal height 643.20255 prevdepth 0.0, prevgraf 4 lines ! OK. l.8 \showlists [1] Output written on xnod.dvi (1 page, 264 bytes). > Submitted: 17:48:09 18 Nov 91 > IPMessageId: A4D0EAFE6C983690 > From: Chris Thompson > To: "Wayne G. Sullivan" > Subject: Re: [xnod] Wayne, Thanks for the messages. There is no doubt that there is a bug: the declaration of t,u,v as |nonnegative_integer|, which I had missed, makes it definite. I have traced through |xn_over_d| using a debugger, and found it storing negative values, as expected. I'll put all this together and send it off to Barbara Beeton, if you are agreeable. Chris Thompson > Accepted: 12:02:21 21 Nov 91 > Submitted: 09:54:31 19 Nov 91 > From: "Wayne G. Sullivan" > To: Chris Thompson > Subject: xn_over_d Thanks in advance for passing the problem on to BNB/DEK. Enclosed below are screen messages which appear when xnod.tex is processed on our ancient VM/CMS version of TeX. Other things happen with the PC version I use, but that uses an assembly language equivalent of the old xn_over_d which will have to be upgraded to whatever Knuth decides to the procedure. It is unlikely that he will employ the obvious modification. [ dek: ^^^^^^^^^^^^^^^^^^^^ which is? ] Ready; T=0.05/0.11 09:49:43 tex xnod This is TeX, VM/CMS Version 2.1 (no format preloaded) ** (xnod.tex.* > 10.00002pt. l.4 \showthe\fontdimen6 \tensy > -10.0pt. l.6 \showthe\fontdimen6 \tensy AMPX031E Low bound checking error TRACE BACK OF CALLED ROUTINES ROUTINE STMT AT ADDRESS IN MODULE XNOVERD 6 024014 TEX MATHGLUE 4 03E1B8 TEX MLISTTOHLIST 73 041F54 TEX AFTERMATH 68 052EBE TEX MAINCONTROL 128 058098 TEX 110 05B86C TEX PASCAL/VS 05BDAE AMPX031E Low bound checking error TRACE BACK OF CALLED ROUTINES ROUTINE STMT AT ADDRESS IN MODULE XNOVERD 10 024074 TEX MATHGLUE 4 03E1B8 TEX MLISTTOHLIST 73 041F54 TEX AFTERMATH 68 052EBE TEX MAINCONTROL 128 058098 TEX 110 05B86C TEX PASCAL/VS 05BDAE AMPX031E Low bound checking error TRACE BACK OF CALLED ROUTINES ROUTINE STMT AT ADDRESS IN MODULE XNOVERD 6 024014 TEX MATHGLUE 7 03E236 TEX MLISTTOHLIST 73 041F54 TEX AFTERMATH 68 052EBE TEX MAINCONTROL 128 058098 TEX 110 05B86C TEX PASCAL/VS 05BDAE AMPX031E Low bound checking error TRACE BACK OF CALLED ROUTINES ROUTINE STMT AT ADDRESS IN MODULE XNOVERD 10 024074 TEX MATHGLUE 7 03E236 TEX MLISTTOHLIST 73 041F54 TEX AFTERMATH 68 052EBE TEX MAINCONTROL 128 058098 TEX 110 05B86C TEX PASCAL/VS 05BDAE ! OK (see the transcript file). l.8 \showlists [1] (see the transcript file for additional information) Output written on xnod.dvi (1 page, 264 bytes). Transcript written on xnod.texlog. ----- End of Included messages ----- Chris Thompson JANET: cet1@uk.ac.cam.phx Internet: cet1@phx.cam.ac.uk <<< end positive/negative xn_over_d, tex.web section 107 ######################################################################## %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Character code reference %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Upper case letters: ABCDEFGHIJKLMNOPQRSTUVWXYZ % Lower case letters: abcdefghijklmnopqrstuvwxyz % Digits: 0123456789 % Square, curly, angle braces, parentheses: [] {} <> () % Backslash, slash, vertical bar: \ / | % Punctuation: . ? ! , : ; % Underscore, hyphen, equals sign: _ - = % Quotes--right left double: ' ` " %"at", "number" "dollar", "percent", "and": @ # $ % & % "hat", "star", "plus", "tilde": ^ * + ~ % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% [ end of message 037 ]