BlackBox Framework Center

Ivan Denisov wrote:Luowy, your new version based on Josef's also fail to detect bad-format string: 0EDX 0A1X 8CX 0EDX 0BEX 0B4X 0X

	PROCEDURE Utf8ToString* (IN in: ARRAY OF SHORTCHAR; OUT out: ARRAY OF CHAR; OUT res: INTEGER);
		VAR i, j, val, max: INTEGER; ch, ch0: SHORTCHAR;
	BEGIN
		ch := in[0]; i := 1; j := 0; max := LEN(out) - 1;
		WHILE (ch # 0X) & (j < max) DO
			IF ch < 80X THEN           (*1 byte   00-7F *)
				out[j] := ch; INC(j)
			ELSIF ch < 0E0X THEN  (* 2 bytes  C2-DF UTF8Tail *)
				val := ORD(ch) - 192; IF val < 2 (*0*) THEN out := ""; res := 2 RETURN END; 
				ch := in[i]; INC(i); IF (ch < 80X) OR (ch >= 0E0X) THEN out := ""; res := 2 RETURN END;
				val := val * 64 + ORD(ch) - 128;
				out[j] := CHR(val); INC(j)
			ELSIF ch < 0F0X THEN  (* 3 bytes 1110xxxx 10xxxxxx 10xxxxxx *)
				val := ORD(ch) - 224; ch0 := ch; ch := in[i]; INC(i); 
				IF (ch0 = 0E0X)&(ch >= 0A0X)&(ch <= 0BFX) OR (ch0 = 0EDX)& (ch >= 80X)&(ch <= 9FX) 
					OR (ch0#0E0X)&(ch0#0EDX)&(ch >= 80X)&(ch <= 0BFX) THEN val := val * 64 + ORD(ch) - 128;
				ELSE out := ""; res := 2 RETURN 
				END;
				ch := in[i]; INC(i); IF (ch < 80X) OR (ch >= 0E0X) THEN out := ""; res := 2 RETURN END;
				val := val * 64 + ORD(ch) - 128;
				out[j] := CHR(val); INC(j)
			ELSE(* 4 bytes *)
				out := ""; res := 2 RETURN
			END;
			ch := in[i]; INC(i)
		END;
		out[j] := 0X;
		IF ch = 0X THEN res := 0 (*ok*) ELSE res := 1 (*truncated*) END
	END Utf8ToString;
	

DGDanforth wrote: Are we agreed that the choice is between luowys (which version?) and Josef's?

DGDanforth wrote:

Ivan Denisov wrote:Please, Doug, make voting for choose between luowy and Josef T. versions.

The draft of the options is:

- we should adopt Josef 15% faster version of Utf8 converter with simple format check
- we should adopt WenYing version of Utf8 converter with well format check according Unicode 7.0 standard

Are we agreed that the choice is between luowys (which version?) and Josef's?

Ivan Denisov wrote:Now, do we need to make voting for adopting this feature in general?
The previous voting was for choosing the encoder realization and not for adopting the Unicode for Component Pascal identifiers. Am I wrong?

Josef Templ wrote:I have committed the latest changes to the issue-#19 branch.
- Utf8-conversion without checking for illegal Unicode characters
- luowy's optimzation for error detection in 3-byte sequences
- Helmut's change of Kernel.LoaderHook.ThisMod parameter to ARRAY OF CHAR because
this eliminates several Utf8-conversions
The branch is in good shape by now and from my point of view ready for merging.

Ivan Denisov wrote:Now, do we need to make voting for adopting this feature in general?
The previous voting was for choosing the encoder realization and not for adopting the Unicode for Component Pascal identifiers. Am I wrong?

Right, we need an extra voting. Hopefully a fast one.

- Josef

Ivan Denisov wrote:Doug, please, make voting!

The version to try is here:
blackbox-1.7-a1.028.zip
blackbox-1.7-a1.028-setup.exe

DGDanforth wrote:I am creating this poll simply trusting and hoping that you all know what you are doing.

Ivan Denisov wrote:We Just choosing between realization of Utf8ToString converter in Kernel.