[Lazarus] Convert record to JSON?

[Bo Berglund via Lazarus]

I have an application that manages the configuration of an embedded
device, which communicates over TCP/IP. Currently I have implmented a
protocol with a command identifier for each data item and the way to
send the data is by sending a packet with the command ID plus the data
to the open socket.

This is less than optimal but was done when the config item number was
low, but it seems to increase over time so I want to find a better
way.

I have just looked into fpJSON a bit (read the wiki but no coding
yet).

I have this question:
The config data are stored in a record and consists of a mix of data
types like strings and numbers as well as booleans.
Is there a simple way to encode the record as a JSON message to be
sent to the device with all of the information in one packet?
If so how would one go about coding that transformation?

If possible I would like it to be transparent such that if the record
is extended later it would automatically include the new members?

[AlexeyT via Lazarus]

Post by Bo Berglund via Lazarus
The config data are stored in a record and consists of a mix of data
types like strings and numbers as well as booleans.
Is there a simple way to encode the record as a JSON message

Yes, it is. You can write to json values by path: /dir1/dir2/dir3/key,

so for your record, make string paths for your values: /myrec/keybool ;
/myrec/keyint ; ...

[Bo Berglund via Lazarus]

On Fri, 20 Jul 2018 20:09:25 +0300, AlexeyT via Lazarus

Post by AlexeyT via Lazarus

Post by Bo Berglund via Lazarus
The config data are stored in a record and consists of a mix of data
types like strings and numbers as well as booleans.
Is there a simple way to encode the record as a JSON message

Yes, it is. You can write to json values by path: /dir1/dir2/dir3/key,
so for your record, make string paths for your values: /myrec/keybool ;
/myrec/keyint ; ...

I am not sure I understand your suggestion...

I was thinking something like this (based on an example in the wiki
(http://wiki.freepascal.org/Streaming_JSON):

type TMyRecord = record
checksum: word;
ssid: AnsiString;
passwd: AnsiString;
macaddr: AnsiString;
addr: TIpAddress;
baud: integer;
tcpport: word;
mode: byte;
channel: byte;
hidden: byte; // hidden (1) or visible (0)
fixedaddress: byte; //Station mode fixed addr instead of DHCP
numsensors: byte; //Number of active DHT sensors (0..3)
dhtinterval: word; // Interval between DHT sensor readings (min)
host: AnsiString;
reserved: array[0..18] of byte;
end;


function RecToJSON(RC: TMyrecord): string;
var
JS: TJSONStreamer;
begin
JS := TJSONStreamer.Create(NIL);
try
JS.Options := JS.Options + [jsoTStringsAsArray];
Result := JS.RecordToJSONString(RC); //Seems not to exist...
finally
JS.Free;
end;
end;

So I have two issues here:
1) It seems like there is no RecordToJSONString method only
ObjectToJSONString.

2) Do I have to convert my record to an object just to be able to use
JSON with it?

I want to avoid having to maintain a function where every member of
the record will be individually handled by name, then I don't gain
anything by going for JSON regarding future additions.
I want the output of the RecToJSON function to contain the field names
and current values of the record without ever changing this function
in the future when new fields are added....

Is this at all possible?

[Michael Van Canneyt via Lazarus]

Post by Bo Berglund via Lazarus
On Fri, 20 Jul 2018 20:09:25 +0300, AlexeyT via Lazarus

Post by AlexeyT via Lazarus

Post by Bo Berglund via Lazarus
The config data are stored in a record and consists of a mix of data
types like strings and numbers as well as booleans.
Is there a simple way to encode the record as a JSON message

Yes, it is. You can write to json values by path: /dir1/dir2/dir3/key,
so for your record, make string paths for your values: /myrec/keybool ;
/myrec/keyint ; ...

I am not sure I understand your suggestion...
I was thinking something like this (based on an example in the wiki
type TMyRecord = record
checksum: word;
ssid: AnsiString;
passwd: AnsiString;
macaddr: AnsiString;
addr: TIpAddress;
baud: integer;
tcpport: word;
mode: byte;
channel: byte;
hidden: byte; // hidden (1) or visible (0)
fixedaddress: byte; //Station mode fixed addr instead of DHCP
numsensors: byte; //Number of active DHT sensors (0..3)
dhtinterval: word; // Interval between DHT sensor readings (min)
host: AnsiString;
reserved: array[0..18] of byte;
end;
function RecToJSON(RC: TMyrecord): string;
var
JS: TJSONStreamer;
begin
JS := TJSONStreamer.Create(NIL);
try
JS.Options := JS.Options + [jsoTStringsAsArray];
Result := JS.RecordToJSONString(RC); //Seems not to exist...
finally
JS.Free;
end;
end;
1) It seems like there is no RecordToJSONString method only
ObjectToJSONString.
2) Do I have to convert my record to an object just to be able to use
JSON with it?
I want to avoid having to maintain a function where every member of
the record will be individually handled by name, then I don't gain
anything by going for JSON regarding future additions.
I want the output of the RecToJSON function to contain the field names
and current values of the record without ever changing this function
in the future when new fields are added....
Is this at all possible?

For classes it is possible, but not for records (yet).
See the jsonrtti unit.

Michael.
--

[AlexeyT via Lazarus]

Post by Bo Berglund via Lazarus
1) It seems like there is no RecordToJSONString method only
ObjectToJSONString.

No function; I was suggesting to make it (func) to your record type
(each rec type has its new function). So record type will convert to

{

  "myrec": {

     "int_var": 10,

    "bool_var": False,

    "str_var": "ddd"

   }

}

[Bo Berglund via Lazarus]

On Fri, 20 Jul 2018 19:33:19 +0200, Bo Berglund via Lazarus
<***@lists.lazarus-ide.org> wrote:

I changed my code now so my recrd is instead an object:

type TMyRecord = Class(TObject)
checksum: word;
ssid: AnsiString;
passwd: AnsiString;
macaddr: AnsiString;
addr: TIpAddress;
baud: integer;
tcpport: word;
mode: byte;
channel: byte;
hidden: byte; // hidden (1) or visible (0)
fixedaddress: byte; //Station mode fixed addr instead of DHCP
numsensors: byte; //Number of active DHT sensors (0..3)
dhtinterval: word; // Interval between DHT sensor readings (min)
host: AnsiString;
end;

I also changed my function to create the JSON string:

function RecToJSON(RC: TMyrecord): string;
var
JS: TJSONStreamer;
begin
JS := TJSONStreamer.Create(NIL);
try
JS.Options := JS.Options + [jsoTStringsAsArray];
Result := JS.ObjectToJSONString(RC);
finally
JS.Free;
end;
end;

When I run this code in my existing application I can step into the
line
Result := JS.ObjectToJSONString(RC);
and hover the mouse over RC and it displays the correct values for all
fields of the object.

But when the line executes the result is a string with only this empty
content:

{}

Why is this?
What have I done wrong?

[Michael Van Canneyt via Lazarus]

Post by Bo Berglund via Lazarus
On Fri, 20 Jul 2018 19:33:19 +0200, Bo Berglund via Lazarus
type TMyRecord = Class(TObject)
checksum: word;
ssid: AnsiString;
passwd: AnsiString;
macaddr: AnsiString;
addr: TIpAddress;
baud: integer;
tcpport: word;
mode: byte;
channel: byte;
hidden: byte; // hidden (1) or visible (0)
fixedaddress: byte; //Station mode fixed addr instead of DHCP
numsensors: byte; //Number of active DHT sensors (0..3)
dhtinterval: word; // Interval between DHT sensor readings (min)
host: AnsiString;
end;
function RecToJSON(RC: TMyrecord): string;
var
JS: TJSONStreamer;
begin
JS := TJSONStreamer.Create(NIL);
try
JS.Options := JS.Options + [jsoTStringsAsArray];
Result := JS.ObjectToJSONString(RC);
finally
JS.Free;
end;
end;
When I run this code in my existing application I can step into the
line
Result := JS.ObjectToJSONString(RC);
and hover the mouse over RC and it displays the correct values for all
fields of the object.
But when the line executes the result is a string with only this empty
{}
Why is this?
What have I done wrong?

Because RTTI is made only for published properties.

So, you must make it published properties:

Type

{ needed if you want to descend from TObject.
You can also descend from TPersistent. }

{$M+}
TMyRecord = Class(TObject)
Published
Property checksum: word Read FChecksum Write FCheckSum;
// etc.
end;

Michael.
--

[Bo Berglund via Lazarus]

On Sat, 21 Jul 2018 09:29:29 +0200 (CEST), Michael Van Canneyt via

Post by Michael Van Canneyt via Lazarus

Post by Bo Berglund via Lazarus
What have I done wrong?

Because RTTI is made only for published properties.
Type
{ needed if you want to descend from TObject.
You can also descend from TPersistent. }
{$M+}
TMyRecord = Class(TObject)
Published
Property checksum: word Read FChecksum Write FCheckSum;
// etc.
end;

I changed my definition as above:

{$M+}
TEspConfiguration = Class(TObject) //object
published
checksum: word;
ssid: AnsiString;
passwd: AnsiString;
macaddr: AnsiString;
addr: TIpAddress;
baud: integer;
....
end;


but then when I compile I get this error (14 times):

wificommhandler.pas(81,5) Error: Symbol cannot be published, can be
only a class

Do I need to declare the fields as private first so they can later be
declared published?

Or are these fields (simple variables and ansistrings) incompatible
with the fpc JSON handling? (Must be a class...)

Maybe it is simpler after all to code this without using fpcjson.
The JSON structure is not that complicated to handle, especially for
writing....

[Sven Barth via Lazarus]

Post by Bo Berglund via Lazarus
On Sat, 21 Jul 2018 09:29:29 +0200 (CEST), Michael Van Canneyt via

Post by Michael Van Canneyt via Lazarus

Post by Bo Berglund via Lazarus
What have I done wrong?

Because RTTI is made only for published properties.
Type
{ needed if you want to descend from TObject.
You can also descend from TPersistent. }
{$M+}
TMyRecord = Class(TObject)
Published
Property checksum: word Read FChecksum Write FCheckSum;
// etc.
end;

{$M+}
TEspConfiguration = Class(TObject) //object
published
checksum: word;
ssid: AnsiString;
passwd: AnsiString;
macaddr: AnsiString;
addr: TIpAddress;
baud: integer;
....
end;
wificommhandler.pas(81,5) Error: Symbol cannot be published, can be
only a class
Do I need to declare the fields as private first so they can later be
declared published?
Or are these fields (simple variables and ansistrings) incompatible
with the fpc JSON handling? (Must be a class...)
Maybe it is simpler after all to code this without using fpcjson.
The JSON structure is not that complicated to handle, especially for
writing....

Look at the code that Michael wrote. He used properties on purpose,
because only *properties* can be published while not being a class type.
Also even if you don't use ObjectToJSONString you can still use fpJSON
as that deals with the whole writing/reading part. You only need to
serialize/deserialize your record manually.

That said please find attached a unit that serializes (nearly) any
record to JSON and reads it back again. It might be buggy, but a quick
test with your example record worked. It's compatible with both 3.0.4 as
well as 3.1.1. Variant records work correctly as well as thankfully
managed types can't be used with them thus simply writing/reading the
memory multiple times works, at least as long as no one modified the
streamed data. :P

As it's two class helpers you can use them like this (Note: for the test
I implemented an "=" operator overload for your record):

=== code begin ===

uses
  fpjson, fpjson.helper, fpjsonrtti;

var
  streamer: TJSONStreamer;
  destreamer: TJSONDeStreamer;
  s: TJSONStringType;
  t1, t2: TMyRecord;
begin
  // init t1
  // ...
  streamer := TJSONStreamer.Create(Nil);
  try
    s := streamer.RecordToJSONString(@t1, TypeInfo(t1));
    Writeln('Serialized record:');
    Writeln(s);
  finally
    streamer.Free;
  end;

  Writeln;

  destreamer := TJSONDeStreamer.Create(Nil);
  try
    destreamer.JSONToRecord(s, @t2, TypeInfo(t2));
    Writeln('Equal deserialized records: ', BoolToStr(t1 = t2, True));
  finally
    destreamer.Free;
  end;
end.

=== code end ===

The output then looks like this:

=== output begin ===

Serialized record:
{ "Field1" : 17185, "Field2" : "Blubb", "Field3" : "Bla", "Field4" :
"54:43:67:94:30:59", "Field5" : { "Field1" : 127, "Field2" : 0, "Field3"
: 0, "Field4" : 1, "Field5" : 16777343 }, "Field6" : 115200, "Field7" :
80, "Field8" : 66, "Field9" : 2, "Field10" : 1, "Field11" : 3, "Field12"
: 28, "Field13" : 21044, "Field14" : "World", "Field15" : [0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }

Equal deserialized records: True

=== output end ===

Note: As the RTTI for records (currently) does not contain the field
names index values are used.

Generic methods in trunk can even improve this a bit more (at least once
I've fixed the bug I've found there):

=== code begin ===

s := streamer.RecordToJSONString(@t1, TypeInfo(t1));
// becomes
s := streamer.specialize RecordToJSONString<TMyRecord>(t1);

destreamer.JSONToRecord(s, @t2, TypeInfo(t2));
// becomes
destreamer.specialize JSONToRecord<TMyRecord>(s, t2);

=== code end ===

@Michael: maybe we can integrate this in fpjsonrtti directly?

Regards,
Sven

[Michael Van Canneyt via Lazarus]

Post by Sven Barth via Lazarus
=== code begin ===
// becomes
s := streamer.specialize RecordToJSONString<TMyRecord>(t1);
// becomes
destreamer.specialize JSONToRecord<TMyRecord>(s, t2);
=== code end ===
@Michael: maybe we can integrate this in fpjsonrtti directly?

:-) :-) :-)

I am glad you proposed it, saves me the trouble of asking it.
I was planning to extend JSONRTTI so it can handle more cases
(I want to replace the rest JSON handling with it),
so this can definitely be included.

I'll study your code and get back to you.
Thanks for the sample code !! :)

Michael.
--

[Sven Barth via Lazarus]

Post by Michael Van Canneyt via Lazarus

Post by Sven Barth via Lazarus
=== code begin ===
// becomes
s := streamer.specialize RecordToJSONString<TMyRecord>(t1);
// becomes
destreamer.specialize JSONToRecord<TMyRecord>(s, t2);
=== code end ===
@Michael: maybe we can integrate this in fpjsonrtti directly?

:-) :-) :-)
I am glad you proposed it, saves me the trouble of asking it. I was
planning to extend JSONRTTI so it can handle more cases (I want to
replace the rest JSON handling with it), so this can definitely be
included.
I'll study your code and get back to you. Thanks for the sample code
!! :)

There are definitely still some problems with it, e.g. a field of type
TObject (or any descendant) currently can't be deserialized as I didn't
want to rewrite DeStreamClassProperty. Then there are the sets which for
non-published properties can be greater than 32-bit (up to 256-bit
currently). Support for dynamic arrays could be added as well.
Also I didn't check whether all types serialize/deserialize correctly,
e.g. especially the special floating point types Comp and Currency.
You'd need to check what Get-/SetFloatProp are doing there.

Also I noticed a problem that's also in the TObject based streaming
code: WideChar and UnicodeChar are deserialized incorrectly as
TJSONStringType is of type UTF8String and thus js[1] might be an invalid
character if it's Codepoint happens to be > $7f.

Regards,
Sven
--

[Bo Berglund via Lazarus]

On Sat, 21 Jul 2018 17:47:54 +0200, Sven Barth via Lazarus

Post by Sven Barth via Lazarus
Look at the code that Michael wrote. He used properties on purpose,
because only *properties* can be published while not being a class type.

Sorry, did not fully note the property keyword!
Not doing that made it easier to support both object and record
through conditional defines without changing the body of the code...

But it now means I have to use a new structure like this:

TIpAddress = array[0..3] of byte;

TEspConfiguration = Class(TObject)
private
Fchecksum: word;
Fssid: AnsiString;
Fpasswd: AnsiString;
Fmacaddr: AnsiString;
Fhost: AnsiString;
Faddr: TIpAddress;
Fbaud: integer;
Ftcpport: word;
Fmode: byte;
Fchannel: byte;
Fhidden: byte;
Ffixedaddress: byte;
Fnumsensors: byte;
Fdhtinterval: word;
published
property checksum: word read Fchecksum write Fchecksum;
property ssid: AnsiString read Fssid write Fssid ;
property passwd: AnsiString read Fpasswd write Fpasswd;
property macaddr: AnsiString read Fmacaddr write Fmacaddr;
property host: AnsiString read Fhost write Fhost;
property addr: TIpAddress read Faddr write Faddr;
property baud: integer read Fbaud write Fbaud;
property tcpport: word read Ftcpport write Ftcpport;
property mode: byte read Fmode write Fmode;
property channel: byte read Fchannel write Fchannel;
property hidden: byte read Fhidden write Fhidden;
property fixedaddress: byte read Ffixedaddress write Ffixedaddress;
property numsensors: byte read Fnumsensors write Fnumsensors;
property dhtinterval: word read Fdhtinterval write Fdhtinterval;
end;

Can the JSON handlers deal with the array type TIpAddress?

And how do I "plug" the fpjson.helper.pp code into my project?

[Bo Berglund via Lazarus]

On Sun, 22 Jul 2018 00:22:58 +0200, Bo Berglund via Lazarus

Post by Bo Berglund via Lazarus
Can the JSON handlers deal with the array type TIpAddress?

Well, it trurns out that this fails earlier on when compiling...
So I have:

type
TIpAddress = array[0..3] of byte;

{$M+}
TEspConfiguration = Class(TObject)
private
Fchecksum: word;
Fssid: AnsiString;
...
Faddr: TIpAddress; //<==
...
published
property checksum: word read Fchecksum write Fchecksum;
property ssid: AnsiString read Fssid write Fssid ;
...
property addr: TIpAddress read Faddr write Faddr; // <==
...
end

When compiling I get the following error on the "property addr" line:

wificommhandler.pas(97,33) Error: This kind of property cannot be
published

If I replace the type declaration TIpAddress with array[0..3] of byte
it just adds another error message...

Can published properties only be simple variable types not including
arrays?
Seems odd since an AnsiString, which is accepted, is just an array of
AnsiChar.

[Michael Van Canneyt via Lazarus]

Post by Bo Berglund via Lazarus
On Sun, 22 Jul 2018 00:22:58 +0200, Bo Berglund via Lazarus

Post by Bo Berglund via Lazarus
Can the JSON handlers deal with the array type TIpAddress?

Well, it trurns out that this fails earlier on when compiling...
type
TIpAddress = array[0..3] of byte;
{$M+}
TEspConfiguration = Class(TObject)
private
Fchecksum: word;
Fssid: AnsiString;
...
Faddr: TIpAddress; //<==
...
published
property checksum: word read Fchecksum write Fchecksum;
property ssid: AnsiString read Fssid write Fssid ;
...
property addr: TIpAddress read Faddr write Faddr; // <==
...
end
wificommhandler.pas(97,33) Error: This kind of property cannot be
published

Only dynamic arrays can be published.

Post by Bo Berglund via Lazarus
If I replace the type declaration TIpAddress with array[0..3] of byte
it just adds another error message...
Can published properties only be simple variable types not including
arrays?
Seems odd since an AnsiString, which is accepted, is just an array of
AnsiChar.

Yes, but it is a dynamic array.

Michael.
--

[Bo Berglund via Lazarus]

On Sun, 22 Jul 2018 07:52:40 +0200 (CEST), Michael Van Canneyt via

Post by Michael Van Canneyt via Lazarus
Only dynamic arrays can be published.

So I changed the declaration:

TIpAddress = array of byte;

Then added a constructor to the class where I set the length to 4:

constructor TEspConfiguration.Create;
begin
SetLength(Faddr,4);
end;

This removed the errors I had before but popped up another one in my
main code:

String2IP(FConfRec.addr, edWiFiAddress.Text); //<== Error here

formmainconfig.pas(240,26) Error: Can't take the address of constant
expressions

This function converts an IP address as string into the byte array.
The data comes from an editbox where the user enters the IP address
required:

function TfrmMainConfig.String2IP(var IP: TIpAddress; Addr: string):
boolean;
var
Lst: TStringList;
i: integer;
t: integer;
begin
Result := false;
Lst := TStringList.Create;
try
Lst.StrictDelimiter := true;
Lst.Delimiter := '.';
Lst.DelimitedText := Addr;
if Lst.Count <> 4 then exit; //First requirement, 4 positions
for i := 0 to 3 do
begin
t := StrToIntDef(Lst[i],-1);
if (t<0) and (t>255) then exit; //Second, must be byte sized
IP[i] := t and $FF;
end;
Result := true;
finally
Lst.Free;
end;
end;

So now it looks like I am stuck between two exclusive requirements...

[Michael Van Canneyt via Lazarus]

Post by Bo Berglund via Lazarus
On Sun, 22 Jul 2018 07:52:40 +0200 (CEST), Michael Van Canneyt via

Post by Michael Van Canneyt via Lazarus
Only dynamic arrays can be published.

TIpAddress = array of byte;
constructor TEspConfiguration.Create;
begin
SetLength(Faddr,4);
end;
This removed the errors I had before but popped up another one in my
String2IP(FConfRec.addr, edWiFiAddress.Text); //<== Error here
formmainconfig.pas(240,26) Error: Can't take the address of constant
expressions

That is because you cannot pass the array as a var argument. Just remove the
var, for a dynamic array it is not required anyway.

Michael.
--

[Bo Berglund via Lazarus]

On Sun, 22 Jul 2018 10:41:23 +0200 (CEST), Michael Van Canneyt via

Post by Michael Van Canneyt via Lazarus
That is because you cannot pass the array as a var argument. Just remove the
var, for a dynamic array it is not required anyway.

I did so and the compile error disappeared.
However, now that there are no compile time errors instead I get a
runtime exception inside the call to ObjectToJSONString here:

function TConfigCommHandler.ConfigAsJSON(RC: TEspConfiguration):
string;
{Create a JSON verson of the configuration record}
var
JS: TJSONStreamer;
begin
JS := TJSONStreamer.Create(NIL);
try
JS.Options := JS.Options + [jsoTStringsAsArray];
Result := JS.ObjectToJSONString(RC);
finally
JS.Free;
end;
end;

The Exception message is:
' : Unsupported property kind for property: "addr"'

So it is still not supported....

[Sven Barth via Lazarus]

Post by Bo Berglund via Lazarus
And how do I "plug" the fpjson.helper.pp code into my project?

Simply add the unit to your project and use it where you use fpjsonrtti (in
addition to that unit!). The methods are then part of TJSONStreamer and
TJSONDeStreamer.

The TIpAddress should work with that code as well.

Regards,
Sven