Next: Foreign Functions, Previous: Knit Functions, Up: k9: Manual
This chapter covers reading and writing to file (File I/O), interprocess communication (IPC), and working with pre-built shared libraries (csv, json, lz4, and zstd).
i/o(*enterprise) 0: r/w line 1: r/w char *2: r/w data *3: k-ipc set *4: https get 5: ffi/import `csv?`csv t;`json?`json t `lz4?`lz4 t;`zstd?`zstd t
k9 reads and writes to files in three different formats including line (list of strings), char (list of characters), and data (k9 format).
Let’s begin with a simple example to creat a sample table, write to csv and then read it back in again. We’ll use a function so the table can easily grow to a larger size.
g:{+`s`f`i!(x?"abc";x?10.;x?10)} / generate function for table
g[5] / generate table with 5 rows
s f i
- ---------- -
c 1.296544 4
c 0.03771765 7
c 3.371475 0
b 1.352739 4
b 1.187619 5
`csv@g[5] / convert table to csv string
"s,f,i\nb,1.08707,3\na,5.506882,0\nc,8.938667,1\nc,6.217895,6\nb,1.542842,6\n"
"sample.csv"1:`csv@g[5] / write to sample.csv file
"sample.csv"
1:"sample.csv" / read from file
"s,f,i\na,3.669518,1\na,0.001226037,7\nb,2.792163,3\nc,8.539221,7\nb,9.333188,7\n"
`csv?1:"sample.csv" / read from file and convert from csv
s f i
- ----------- -
a 3.669518 1
a 0.001226037 7
b 2.792163 3
c 8.539221 7
b 9.333188 7
Given the small size of the file all this happens too quickly to notice the speed. Let’s give it a go with a bigger file.
\t "sample.csv"1:`csv@g[_1e7] 2587 \t t:`csv?1:"sample.csv" 388
Ten million rows via csv in under 1/2 a second. Of course results will depend on hardware and number of columns also.
Read from file x.
0:"some.csv" a,b 1,3. 2,4.
Output to string x (file name or null for stdout) the list of strings in y. y must be a list of strings. If y is a single string then convert to list via list.
""0:("blue";"red") / "" represents stdout
blue
red
"file.txt" 0: ("blue"; "red") / write to file, one line per element
Read in file x (string).
/ file.bin contains the binary value 0x0010FA37 1:"file.bin" 0010fa37 / file.txt contains the ascii string 37\ntest\n 1:"file.txt" "37\ntest\n"
Write to file x (string) data y (list of characters).
"some.txt"1:"0123ABab" "some.txt" 1:"some.txt" "0123ABab" "some.csv" 1:`csv [[]a:1 2;b:3. 4.] "some.csv" 1: "some.csv" "a,b\n1,3.00\n2,4.00\n" `csv?1:"some.csv" a b - - 1 3 2 4
Enterprise Only
Load file, eg. csv or from a (x 2: y) save. For the latter, one can find a “save then load” example in the next section.
2:`t.csv s t e p z ---- -------- - -- ---- AABL 09:30:00 D 11 4379 AABL 09:30:00 B 40 3950 2:`r / read from file a b c d e ---------- ---------- ---------- ---------- ---------- 0.5366064 0.8250996 0.8978589 0.4895149 0.6811532 0.1653467 0.05017282 0.4831432 0.4657975 0.4434603 0.08842649 0.8885677 0.23108 0.3336785 0.6270692 ..
Enterprise only
Save to file x non-atomic data y (e.g., lists, dictionaries, or tables).
This example saves 100 million 8-byte doubles to file. The session is then closed and a fresh session reads in the file. Both the write (420 ms) and compute statistics from the file have impressive speeds (146 ms) given the file size (800 MB).
n:_1e8 r:+`a`b`c`d`e!5^n?1.;r `r 2:r / write to file
Start new session.
\t r:2:`r;select avg a,sum b, max c, min d, sum e from r 148
k9 has the ability to comunicate between separate k9 instances via interprocess communication (IPC). These processes can be on the same or separate machines. A user will start multiple instances specifying the port (via the command line argurment -p) that the k9 session will use, opening a handle to that port, and then running remote commands via the handle and 3: or 4:. The commands to be run are sent as a string, eg. “3+12”.
Open a connection to port x and return a handle. If the port has been forwarded from another machine (eg. ssh -L 1280:server.com:1280 laptop.com) then this handle allows remote execution.
/ before running this session start another k session in another / terminal window specifying that port 1251 should be used / k -p 1251 h:3:1251 / create handle to port 1251 and save to h h / h is an integer 4
Exectute string y using handle x. Handle x should have already been created using k-ipc. set is asynchronous, thus once the command is sent the session immediately returns to the user. One will not be able to return any results via set. If a result is required then one should use get.
h:3:1251 / create handle to port 1251 h 3:"a:12" / set a to 12 on remote session h 3:"n:_1e8;b:1?n?1." / set b to a random number on remote
TBD
Execute string y using handle x. Handle x should have already been created using k-ipc. get is synchronous, thus once the command is sent the session waits until there result is ready.
h:3:1251 / create handle to port 1251 h 3:"a:12" / set a to 12 on remote session h 3:"n:_1e8;b:1?n?1." / set b to a random number on remote h 4:"a" 12 h 4:"b" ,0.07820029 a:"local" / set a locally to a string value a "local" 4:"a" 12
These topics will be covered in the Foreign Functions chapter.
Enterprise only
csv tools allows one to export tables into csv and import from csv. Generally these functions are used with file i/o 1:
"test.csv"1:`csv@[[]x:`a`b;y:3 2] / write sample table to file "test.csv" `csv?1:"test.csv" / read csv from file x y - - a 3 b 2
Export object x to json format or import string x from json format. The json shared library will be automatically pulled from the Shakti website into the current directory. Use of symbolic links with path on the binary may cause issues.
"test.json"1:`json@[[]x:`a`b;y:3 2] "test.json" `json?1:"test.json" (,"x";,"y")!(,"a";3) (,"x";,"y")!(,"b";2)
lz4 and zstd are compression algorythms which can be used to reduce the size of files. As with other foreign libraries k9 does the work behind the scenes. Here is an example of writing a large csv file to disk and reading it back in. In the second (lz4) and third (zstd) examples the csv’s are compressed and decompressed with lz4 and zstd respectively. One can look at the file sizes to see how well the compression worked.
g:{+`s`f`i!(x?"abc";x?10.;x?10)}
"t.csv"1:`csv@g[_1e7]
"t.csv"
\t tc:`csv?1:"t.csv" / read from csv
314
"t.lz4"1:`lz4@`csv@g[_1e7] / write to lz4
"t.lz4"
\t tz:`csv?`lz4?1:"t.lz4" / read from lz4 compressed csv
525
"t.zstd"1:`zstd@`csv@g[_1e7] / write to zstd
"t.zstd"
\t tz:`csv?`zstd?1:"t.zstd" / read from zstd compressed csv
535
Next: Foreign Functions, Previous: Knit Functions, Up: k9: Manual